CCD Spectrograph Core - Fast 16-bit

What's new

The CCD Transmission Spectrograph project was cool enough, but it lacked two things - the ability to digitize all 3694 pixels, and the ability to do so at 16-bit resolution. The solution to the first problem was found in the 8-bit fast spectrograph core project. This new project is the solution to both problems - width and depth. An AD7667 16-bit 1 MSPS converter and an ATmega1284 as an Arduino. Using the 16-bit 1MSPS converter it can digitize a frame in 16mS, the same as as the 8-bit fast version.

Hardware

The circuit is built on a 100mm x 100mm PCB, and I used a 48-pin SchmartBoard component carrier for the AD7667. It makes soldering the 48-pin TQFP a little easier than soldering it to the PCB directly. The AD8021 amplifier is soldered on the PC board directly underneath the ADC. There is a power supply that provides digital +5V and analog ±5V from a single 7 to 12VDC input.

This schematic has a different look than those I usually do because this is the "real" schematic from which the board is made. I normally translate the schematic into an easier to read version using software that I don't use for making boards, but this was a little too complicated for me to keep the two schematics in sync.

The ATmega1284 has to generate clocks to drive the CCD and the ADC. On port D there are just enough lines to do that and still have serial communication with the host. Three lines for the CCD, and three lines for the ADC. Timer 2 is used to generate the MCLK signal on pin OC2A so no PWM can be used on Timer 2. There are four more PWM pins on the unused port B.

The logic analyzer screen capture below shows the clocks generated and how they are synchronized with the MCLK. ICG and SH are the other CCD clocks while CNVST, RD, and BYTESWAP are the ADC control lines.

Spectrograph Timing

CCD Sensor

The sensor is a Toshiba TCD1304AP, a 3648 pixel linear CCD sensor which operates on a single voltage (3.0V to 5.5V). The sensor is driven directly by the outputs of the ATmega1284.

The Toshiba TCD1304 data sheet is illegible and incomplete rubbish. It offers no insight into the workings of the device. From poking and prodding this thing I think I have a handle on how it operates. The TCD1304 is always clocking the shift registers if the MCLK is running. It is only when you hit it with the combination of ICG LOW and SH HIGH and then LOW that it dumps the photodiodes onto the shift registers. You then have 14776 MCLK cycles (3694 total pixels) worth of data, and it's back to empty pixels. That is a good thing, though. It keeps dark signal from building up in the output shift registers.

The data sheet shows the SH clock running at 1/2 the pixel rate. Judging by the behavior of the output, I would speculate that the SH clock alternately puts pixel data from the two shift registers on the output buffer's gate. The MCLK determines which pixel pair that would be in the full stream. A complete assumption.

Video Buffer

The range and polarity of the CCD output is wrong for the ADC. The signal starts out at 2.6V and goes toward ground with increasing exposure. At saturation, the output voltage is 480mV. That is a 2.2V signal span between the background noise and saturation. The ADC needs the voltage to start near ground and work its way up to 2.5V max. I say near ground because we need the noise floor to be accessible to the ADC so it can be removed digitally to calibrate the signal.

Analog Devices recommends the AD8021 amplifier, and I see no point in trying to outsmart them. The AD8021 is wired as a unity gain inverting amplifier. The 2.6V offset is removed by supplying the non-inverting input with the adjustable voltage from a pot.

ADC

The ADC is the Analog Devices AD7667 16-bit 1 MSPS Unipolar ADC. It has both serial and parallel interfaces, but I chose parallel for speed. The data is taken from the upper 8 bits using the BYTESWAP signal to select first the low byte and then the high byte. Raw 16-bit data is stored in RAM in the microcontroller for later use.

The input range is 0 to 2.5V, of which we use 2.2V for signal.

There are capacitors all over this circuit. They are necessary to keep the digital noise from getting in the analog circuits and messing up that beautiful 16-bit reading. I used 10uF ceramics, but the board will take more or less. The sensitivity of the 16 bit converter is 2.5V / 65536 = 38µV per ADU. The noise doesn't have to be below that, but the SNR has to be high enough that the signal doesn't get lost.

Microcontroller

The microcontroller is an Atmel ATmega1284. The ATmega1284 has 16kB of RAM, and our video buffer uses just about half of it. The code is very simple, and only uses around 4kB of flash.

The 16MHz ATmega1284 is programmed with an Arduino bootloader and the serial connection allows you to program it from within the Arduino IDE. The CCD, ADC and serial use two full ports, C and D, but ports A and B are available for other uses. That includes four PWM outputs on port B and 8 analog inputs on port A.

Buffer Amp Power Supply

Ok. I originally designed this with the MAX232 as a charge pump, and when I was only powering the opamp, that worked fine. When I started powering all of the analog circuitry from it the MAX232 failed to keep up. I had to redesign the power supply mid-stream. I changed to a 78L05 running from the input supply and a MAX660 running from the digital 5V supply.

The MAX660 is a charge pump power supply that puts out approximately (-)Vin, where Vin is the 5V line. The opamp uses ±5V while the ADC and CCD use +5V. Digital and analog +5V supplies are separate. The analog +5V is derived from the input power source using an 78L05. In my case the input voltage was 5.02V and the output was -4.95V.

Before Using

There is a 2-pin header on the board labeled "CAL". Don't put the jumper on until you have adjusted the pot labeled "OFS" to set the inverting amplifier output offset to just a bit above ground. The pot will allow you to adjust the voltage to just under ground, too, and that would be catastrophic if the jumper was installed. The ADC has protection diodes to short out voltages more than 0.3V below ground. Excessive current would flow. The first casualty would be the very expensive ADC.

In fact, it is a good idea never to adjust the offset pot with the jumper installed.

To adjust the offset, pull the "CAL" jumper if installed and connect an oscilloscope to the pin farthest from the ADC. That is the amplifier output. Adjust the "OFS" pot to set the amplifier lowest output voltage to just over ground by 100mV or so. Then put the jumper on and test.

Microcontroller Firmware

The Arduino spectrograph software is written in C in the Arduino IDE. Line readout consists of driving the CCD and ADC clocks and reading the output of the ADC. With the AD7667 we can comfortably digitize one pixel every 4.5µS. The 888.88kHz MCLK and the pixel read are coordinated to be synchronous. Each pixel takes 4 MCLK cycles, but the MCLK is a free-running clock generated by Timer2, which is synchronized at the start of each line. There are 18 CPU cycles in one MCLK. Not that we care.

The instructions in the "readLine" function are arranged to use exactly the right number of cycles for the MCLK speed. If you speed up the MCLK, and leave the CPU clock the same, you must also remove cycles from this routine. If you slow down the MCLK, you must add cycles here. There are 72 CPU cycles in one pixel time. This we care about. We have to use every one of those cycles during the readout process to keep the clocks from skewing and messing up the association between our pixel number and the physical pixel on the CCD.

#include <util/delay_basic.h>

#define RD (1<<2)
#define CNVST (1<<3)
#define BYTESWAP (1<<4)
#define ICG (1<<5)
#define SH (1<<6)
#define MCLK (1<<7)

// Full frame, including dark pixels
// and dead pixels.
#define PIXEL_COUNT 3691

// Ports and pins
#define CLOCKS PORTD
#define CLOCKP PIND
#define CLOCKS_DDR DDRD
#define DATA_PINS PINC
#define DATA_PORT PORTC
#define DATA_DDR DDRC

// 10mS exposure time.
#define EXPOSURE_TIME 10

// Initial clock state.
uint8_t clocks0 = (RD + CNVST + ICG);

// 16-bit pixel buffer
uint16_t pixBuf[PIXEL_COUNT];

char cmdBuffer[16];
int cmdIndex;
int exposureTime = EXPOSURE_TIME;
int cmdRecvd = 0;

/*
 * readLine() Reads all pixels into a buffer.
 */

void readLine() {
  // Get an 8-bit pointer to the 16-bit buffer.
  uint8_t *buf = (uint8_t *) pixBuf;
  int x = 0;
  uint8_t scratch = 0;
  
  // Disable interrupts or the timer will get us.
  cli();
  
  // Synchronize with MCLK and
  // set ICG low and SH high.
  scratch = CLOCKS;
  scratch &= ~ICG;
  scratch |= SH;
  while(!(CLOCKP & MCLK));
  while((CLOCKP & MCLK));
  TCNT2 = 0;
  _delay_loop_1(1);
  __asm__("nop");
  __asm__("nop");
  __asm__("nop");
  __asm__("nop");
  __asm__("nop");
  CLOCKS = scratch;
  
  // Wait the remainder of 4uS @ 20MHz.
  _delay_loop_1(22);
  __asm__("nop");
  __asm__("nop");

  // Set SH low.
  CLOCKS ^= SH;

  // Wait the reaminder of 4uS.
  _delay_loop_1(23);

  // Start the readout loop at the first pixel.
  CLOCKS |= (RD + CNVST + ICG + BYTESWAP + SH);
  __asm__("nop");
  
  do {
    // Wait a minimum of 250nS for acquisition.
    _delay_loop_1(2);

    // Start the conversion.
    CLOCKS &= ~CNVST;
    CLOCKS |= CNVST;

    // Wait a minimum of 1uS for conversion.
    _delay_loop_1(4);

    // Read the low byte of the result.
    CLOCKS &= ~RD;
    _delay_loop_1(4);

    *buf++ = DATA_PINS;

    // Setup and read the high byte.
    CLOCKS &= ~(BYTESWAP);
    _delay_loop_1(4);

    *buf++ = DATA_PINS;

    // Set the clocks back to idle state
    CLOCKS |= (RD + BYTESWAP);

    // Toggle SH for the next pixel.
    CLOCKS ^= SH;

  } while (++x < PIXEL_COUNT);
  
  sei();
}

/*
 * clearLine() Clears the CCD.
 */

void clearLine() {
  
  int x = 0;

  // Set ICG low.
  CLOCKS &= ~ICG;
  CLOCKS |= SH;
  _delay_loop_1(14);

  // Set SH low.
  CLOCKS ^= SH;
  _delay_loop_1(10);

  // Reset the timer so the edges line up.
  TCNT2 = 0;
  
  CLOCKS |= (RD + CNVST + ICG + BYTESWAP + MCLK);
  
  do {
    CLOCKS ^= SH;
    _delay_loop_1(10);
    
  } while (++x < PIXEL_COUNT);

}

/*
 * sendLine() Send the line of pixels to the user.
 */
void sendLine() {
  uint16_t x;

  for (x = 0; x < PIXEL_COUNT; ++x) {
    Serial.print(x);
    Serial.print(",");
    Serial.print(pixBuf[x]);
    Serial.print("\n");
  }
}

/*
 * setup()
 * Set the data port to input.
 * Set the clock port to output.
 * Start timer2 generating the Mclk signal
 */

void setup() {
  delay(10);
  CLOCKS_DDR = 0xff;
  CLOCKS = 0; //clocks0;
  DATA_DDR = 0x0;
  Serial.begin(115200);

  // Setup timer2 to generate an 888kHz frequency on D10
  TCCR2A = (0 << COM2A1) | (1 << COM2A0) | (1 << WGM21) | (0 << WGM20);
  TCCR2B = (0 << WGM22) | (1 << CS20);

  OCR2A = 8;
  TCNT2 = 0;
  delay(10);
}

/*
 * loop()
 * Read the CCD continuously.
 * Upload to user on switch press.
 */

void loop() {

  int x;
  char ch;

  // If we got a command last time, execute it now.
  if (cmdRecvd) {
    if (cmdBuffer[0] == 'r') {
      
      // Send the readout to the host.
      sendLine();
    } else if (cmdBuffer[0] == 'e') {
      delay(10);
      Serial.write(cmdBuffer);
      // Set the exposure time.
      sscanf(cmdBuffer + 1, "%d", &exposureTime);
      if (exposureTime > 1000) exposureTime = 1000;
      if (exposureTime < 1) exposureTime = 1;
    }
    
    // Get ready for the next command.
    memset(cmdBuffer, 0, sizeof(cmdBuffer));
    cmdIndex = 0;
    cmdRecvd = 0;
  }
  // Clear the CCD.
  clearLine();

  // Integrate.
  delay(exposureTime);

  // Read it for real.
  readLine();

  // See if the host is talking to us.
  if (Serial.available()) {
    ch = Serial.read();

    // If char is linefeed, it is end of command.
    if (ch == 0x0a) {
      cmdBuffer[cmdIndex++] = '\0';
      cmdRecvd = 1;

    // Otherwise it is a command character.
    } else {
      cmdBuffer[cmdIndex++] = ch;
      cmdRecvd = 0;
    }
  }
}

Operation

There are two commands. One, "r\n", causes the spectrograph to spit out a csv organized set of data with pixel number followed by the video data for that pixel. The CCD is continuously read, with the results stored in RAM. When the "r\n" command is received, the data from RAM is shipped serially at 115.2kBaud to the host.

The other command "e<nnn>\n" sets the exposure time in milliseconds, that is "e100\n" sets the exposure time to 100mS.

Host Software

Amplitude Calibration

  • Dark Frame - A frame which contains all signal not generated by light input.
  • Flat Field Frame - A frame which contains only signal generated by light input, normalized to one.
  • Science Frame - The reason you are doing all of this - the final result.

The pixels in a CCD line sensor are photo diodes. They are all created at the same time, and so should be fairly consistent from one to the next. Fairly. Maximum 10% non-uniformity. To do photometry you don't need fair consistency, you need certainty. That is where calibration comes in.

Dark Frames

Standard darks and flats are used to calibrate the amplitude readings. A little explanation is in order. A dark frame is a frame that contains all of the signal present in the output that is not based on light input. You must use the same integration time, readout time, and temperature for each dark, each flat, and each science frame. They individual dark frames are median combined. That gets rid of much of the noise, leaving just the dark signal. You use as many dark frames as practical to generate the master dark frame. The noise goes down as the square root of the number of frames. If Xn is the noise in one frame, the noise in 2 frames is Xn / √2, or Xn / 1.414.

Flat Field Frames

A flat field frame is a frame that shows only the signal that is based on the light input. It is the opposite of a dark frame. You make a flat field by turning on the lamp with no filter in place, and capturing several frames. Each frame has the dark frame subtracted from it, and then the median found as above. The pixel values in the flat field frame are normalized to 1. That is, the highest value is divided into each pixel, resulting in a line of numbers all between zero and one. That is the master flat field frame. "Flat" may be appropriate because if you flat field an empty frame you get a perfectly flat line on the graph.

Making the Final Product

To produce a science frame, you place the transmission filter or other subject on the stage, make a frame, and then process it. To process the frame, you subtract the dark frame from it, then divide each pixel by the corresponding pixel in the flat field frame. The result is a frame that consists only of amplitude corrected light generated signal, and the all of the noise. To cut the noise, you median several science frames, and noise is reduced as in the dark and flat field frames.

Relative vs. Absolute Brightness

You may notice from all of the above that the calibration process gives you a relative brightness level, 0% to 100%. There is nothing absolute involved. That is because what we are measuring, the response of dichroic filters, is a relative thing. A filter passes x% in its passband and y% in its stopband. That is all you need to know about the filter. But what if you need to get a spectrogram from an outside source, like the sodium vapor lamp down the street? You still want relative brightness between places on the spectrum, but the absolute brightness may be anywhere in a large range. To get around that you change the integration time. The default integration time is 10mS. You may set the integration time to be anything from 1 to 1000mS.

When you change the integration time, you need to generate new darks and flats. Keep them around, labeled as dark_001, flat_010, etc. The number is the integration time in mS. When you make a science frame, use the appropriate dark and flat for the integration time you are using.

Wavelength Calibration

This is more tricky. If you know the wavelength at 90° incidence, and the angle of incidence of two wavelengths, you can calculate up an equation to compensate.

Output

Spectrograph Output

The board in action. The CCD is covered with 4 layers of black microfiber cloth and the room is not well lit. The picture was taken with a flash. The scope is on the input to the ADC, and shows the CCD is almost saturated - maybe 90% capacity. The image below is the readout of the CCD with the same setup.

Spectrograph Output

I ordered some holographic diffraction gratings from Edmund to try them out. They were inexpensive at $16.95 for 15. We'll see how well they work. The next better one was $85.

Things I Might Like to Change (already)

  • An FT232H parallel FIFO to USB converter chip. I feel the need for speed after downloading over 6 kB for a single frame. That would provide a single USB connection which could transfer 8MB per second. It would use four more control lines while sharing the data bus with the ADC. Perhaps more appropriate for an area sensor like the KAF-0400, KAF-1600, or KAF-3200.
  • A separate board for the CCD. This would make it easier to embed the device. There is a limit to how long you can make the clock lines, though. Ringing on the clock lines may damage the CCD.
  • Include meta-data in the downloaded frame so automation could select a proper set of calibration frames. The inclusion of the exposure time would allow the program to choose the correct dark frame and flat field frame. The ADC has a very accurate analog temperature sensor in it that could be read by one of the ATmega1284's analog inputs.
  • Automatic exposure control. Before it makes a science frame it checks then adjusts the levels to keep the CCD from blooming.