Is there a new link to this code?

http://www.earthinfo.com/helpfiles/GDdataflags.pdf

I found info on the flags.

The wget program has a recursive mode with lots of nice features, like checking timestamps (can re-start the download partway through the process and skip files already completed). It also has bandwidth management, adjustable random delays between downloads to avoid hammering your link.

My personal choice is perl for data conversion, simply because perl has excellent string manipulation libraries and you end up with a very short conversion program. It isn’t ultra-fast, but all thing considered not too bad. Probably C++ will beat perl for speed if you have a long time to spend tuning the code. Perl will win if programmer time is at a premium and you just do the obvious first level optimizations.

The converted file is large, but that’s in an ASCII format for R to read… internally R will convert to a binary format and allow you to save your workspace on exit. There are some tricks built into R that apply compression (e.g. tokenizing the station ID’s and converting dates to simple integers) so your saved workspace file is much smaller than the original ASCII and loads fast when you go back into the program.

This should run equally well on Microsoft, Linux, Apple, etc. Both perl and R have been widely ported. The Microsoft Windows port of R takes advantage of many of the local windowing features so it “feels” kind of native.

So to ask a question here, I see you cut up the fields into a value plus: MFLAG, QFLAG, SFLAG each of which is a single character. Are those explained anywhere? Looking at your code it seems at first glance that you skip over these flags and don’t do anything with them. Am I right? Or is there something subtle at work here?

Once I get the import working cleanly I’ll put some R scripts together to extract some averages, graphs, etc. Anyone who hasn’t tried R should at least give it a go. Very good for post-processing engineering data, financial data, or scientific plotting.

Cygwin was a lifesaver for me when for the first time in
decades, I found myself at a job with no access to unix or linux,
but windows only. It enabled me to be productive in a way that
would have been nearly impossible without something like Cygwin.

Actually, the crayons have been the province of Hansen, Steig, and Schmidt.

Mann’s notorious by-product was the Hockey Schtick, which included such “delights” as:

– Hide the Decline, in which declining tree-ring proxy data were truncated and replaced with a spliced line of fraudulent temperature numbers, without disclosing this action;

– Spurious principal components analysis in which red noise fed into the code produces a HS every time;

– Plugging Tiljander’s lake varve data into the algorithm upside-down, then denying it, then submitting a letter of retraction in a different journal and falsely claiming to have been exonerated of the charge of using data upside-down;

– Getting journal editors fired if they published, or tried to publish, any work that called his work into question;

– The infamous use of a single, outlier tree series in Yamal, Russia (YAD061) in such way that the entire blade of the HS that is in excess of the Medieval Warm Period was caused by this one tree;

– Eliminating the vast majority of the available 20th Century tree-ring data from his analysis, because they were “not the droids he’s looking for”;

– The continued incorporation of American stripbark Bristlecone pines long after he knew, or should have known, that they were completely useless as a source of temperature signal; and finally, last but not least,

– Sectioning (i.e., cutting down) one of the oldest known living trees in existence on the absurd grounds that he could not work with this tree’s rings by the standard means of taking a sample core via hand drill.

RTF

Mann doesnt use a code. He takes satellite maps and fills in the missing data with red crayons