I know that this has nothing to do with lNet or FPC, but honestly I’ve been having trouble getting the word out for it and I thought putting it on my own blog/page couldn’t hurt.

As for lNet, things are still moving, some small bugfixes have been submitted by people and I’m slowly getting to working on the Mac OS X Carbon/Cocoa visual port. Also, thumbs up to Lazarus guys for the great 1.0 release.

The steps needed for POST to work are roughly:
1. Set HTTP.Method to hmPOST
2. Assign HTTP.OnCanWrite event handler
3. Set HTTP.ContentLength = SizeOf(your POST data)
4. Do HTTP.AddExtraHeader(‘Content-Type: application/x-www-form-urlencoded’); for basic url-encoded POST, or other if you need something else
5. Do HTTP.SendRequest
6. Inside HTTP.OnCanWrite handler, do aSocket.Send or aSocket.SendMessage to send your data
7. Inside HTTP.OnCanWrite handler, change OutputEof to wsPendingData if not all data was sent
OR
7. Inside HTTP.OnCanWrite handler, change OutputEof to wsDone if all data was sent (you need to set it at any rate!)

This should be it. I hope everyone now has a better picture of how to use POST with lNet’s HTTP.

There’s a problem with the logic of FTP.GetData. FTP protocol specifies, that when downloading a file, the DATA connection is established and when the download is complete it is closed.

On socket level (OS level, not lNet), recv() syscall will return 0 when a connection gets closed at the sender. This would normally indicate to us that the FTP file transfer is complete. However, lNet’s sockets return 0 also on “non-errors”.

Non-errors are situations in which the recv() call returns -1, and the error tells lNet that “I would receive some data, but my OS buffer is full, please wait a bit”. lNet handles this internally, and returns 0 on the TLSocket.Get. This is a bit of a problem because in FTP this is returned on GetData and thus a false “file complete” message is sent.

The only proper solution to this problem is reworking of this logic so that GetData returns when the transfer is complete in an exact way. This means that next lNet release a logical change will take place in the FTP client component which might break existing code :(.

It’s missed things like these which make me most sad, because it’s a small problem, but it an have big impact. Until 0.6.6 (or even 0.7.x possibly) is released, the trunk lNet has a workaround in the FTP examples. People who use FTP atm. and have this problem need to check if the file they’re downloading is complete (compare expected size to received size) before closing it and changing state to “file downloaded”, even if FTP.GetData returns 0.

I’m not yet sure how the new GetData will behave. It might be an API change (so old code breaks, and people won’t get silent bugs) for example with a new argument which will tell if the transaction is completed.

Other noteworthy additions in the 0.6.4 release are major fixes:

• Fix compilation of WinCE lNet on fpc 2.4.0+
• Fix FTP visual example (missing file in svn)
• Fix http URL parser (no more trailing / problems)
• Fix FTP bug when retreived file was unavailable from server (hanging socket syndrome)

All in all a nice release. The only thing really missing now is a fully functioning (which means on windows too) IPv6 and Carbon/Cocoa integration in Lazarus.

I’ll also try and get that Qt4 interface done, although this will probably require changes in the libqt4pascal C++ code as well (if anyone is willing to help with this, it’d be a huge help). There have been a lot of questions regarding the working of HTTP code in lNet, mainly GET/POST and some specific header questions. I’ll try and get Micha to write a nice how to on this page.

All in all about 15 bugs got fixed, but lets get it over from the start.

First there was a problem with SMTP when SSL was turned on (either implicitly or via STARTTLS). There was a problem with sending bigger chunks of data, like mails with attachments. It turned out that due to an oversight on my side (albeit the OpenSSL docs aren’t exemplary either), I misunderstood the behavior of SSL_write(). It seems that when SSL_write() fails on non-blocking send with SSL_ERROR_WANT_WRITE error, the next call to the function MUST contain the exact same buffer pointer. This was a cheap check done by OpenSSL to ensure that the data you’re trying to send doesn’t change inbetween the two calls. Problem is, I was using ansistrings which got filled up (and thus relocated). This got promptly fixed by setting the SSL_CTX to ignore that check.

Another bug got reported right after I fixed this part. It seems that I completely forgot to handle the TLConnection.Session property in regards to visual assignment in Lazarus via OI. Not only did I make it so that it crashes on certain changes, I completely forgot to use the TComponent notification mechanism.

While fixing that extempore I found another, bigger one. My Sessions are NOT shared at all! I made a HUGE unforgivable logical oversight which caused each TLSession (and descendants) to only work for the last component they were assigned to. In other words, the sessions couldn’t be shared properly.

While fixing THAT bug, I found out that my TLSocket.Creator property isn’t delegated properly, resulting in wrong assignment. The idea behind .Creator is, that you will always know the highest-most protocol component by it, so you can go down the chain. For example, TLFtp uses TLtcp which makes the TLSockets. On your events you get the aSocket: TLSocket as argument, and if you ask it for it’s .Creator, you should get the FTP component.

So I fixed these three huge oversights, but the finish line was still far away!

I got another bugreport from my friend about SMTP. He wanted to automate mail sending, more precisely send a SMS (via smtp) and a normal e-mail automatically. Since he needed it inside a visual application he followed my visual SMTP example and automated the process. He found out that if the SMTP.PipeLine was turned off (default, means “emulate pipelining on server”), his commands got “stuck” after the first 2.

This was another logical bug on my part. In SMTP and FTP, I have a state-machine which takes care of the various states in these protocols. It is implemented as a stack, I push the status in, wait for server response and then remove it. The problem was, that I reported success or failure via OnSuccess/OnFailure BEFORE I removed the given status from the stack. If you sent another command inside OnSuccess/OnFailure event handler, the command would get pushed on the stack, but when the execution returned to me, removed again. This basically causes a de-sync, because now SMTP/FTP is waiting for a command not issued.

There was one additional bug fixed during the marathon, not reported but noticed while I tested SSL on both SMTP and HTTP examples. In HTTP I noticed that in Windows, HTTPS download of a page failed, it just got stuck in the middle. I was quite sure it had to do with the windows gui eventer I use in case of visual lNet, because it worked in Linux, and the console version worked in Windows too. Since I use WSAAsyncSelect for the visual eventer core functionality, I decided to properly study it’s documentation on MSDN and see what’s up. It hit me when I read about the “re-enabling” functions. You see, in WSAAsyncSelect, you tell windows “watch this socket for event xxx”. When the given event happens (like “can read”), you get a windows message. The problem is, that after it’s reported, WSAAsyncSelect DOESN’T watch for the event anymore, until you call a “re-enabling” function. However with SSL_read, it’s a problem, because it doesn’t call recv() always, it sometimes just processess the rest of its internal buffers. This causes a de-sync of the event flip-flop. I fixed it by a little hack, since I want the event to always be reported if there’s anything in the buffer, I did a recv() with MSG_PEEK after each read event reported. This ensures that WSAAsyncSelect watches for reads again without removing anything from the recv buffer.

So.. that’s about it. There were some minor example fixes along the way but nothing to write about. I hope this doesn’t undermine lNet’s viability for you. I’d also like to thank my friend Jano Kolesár for all the testing on SMTP.

WinCE fpc 2.2.0 doesn’t currently contain an OpenSSL.pas file. It is present physically but not compiled for that platform. I was only testing things on the fixes (fpc 2.2.1) branch and didn’t realize this. The problem is that lNet requires OpenSSL.pas since 0.6.0 (it doesn’t however need openSSL library to be present unless you use SSL stuff).

I’ll fix this by adding OpenSSL.pas into lNet itself until at least fpc 2.4.0 is out, but until then, please follow the workaround I posted in news section.

0.6 saw an addition of sessions and some minor redesigns. All in all I feel good about this release. Apart from the logical problem with SSLSession, all seems to be working ok.

Another thing some of you might have noticed is that the new release didn’t change sending in any way. Indeed, in the past I had it planned to change sending in basic TLSocket to be more “user friendly” by making it buffered and automatic on TCP connections. In other words, if you send something on TCP you no longer have to care about how big it is and if it made it in one send command. The idea to do this was scrapped however, mostly because lNet still is lightweight, and it was decided to stay so. The original design made this change a bit more complicated, and I decided (after some persuation by Micha) that it’s better to leave things as is.

If someone needs to simplify sending of bigger chunks of data, they can always inherit from TLTcp and TL[SSL]Socket and change how sending works. But the basic building blocks of lNet will remain as they are.