A Decade of TTH: Its Selection and Uncertain Future

February 28, 2013 by 10 Comments

NMDC and ADC rely on the Tiger Tree Hash to identify files. DC requires a cryptographic hash function to avoid the previous morass of pervasive similar, but not identical, files. A bare cryptographic hash primitive such as SHA-1 did not suffice because not only did the files need identification as a whole but in separate parts, allowing reliable resuming and multi-source downloading, and per-segment integrity verification (RevConnect unsuccessfully attempted to reliably use multi-source downloading precisely because it could not rely on cryptographic hashes).

Looking for inspiration from other P2P software, I found that BitTorrent used (and uses) piecewise SHA-1 with per-torrent segment sizes. Since the DC share model asks that same hash function work across entire shares, this does not work. eDonkey2000 and eMule, with per-user shares similar to those of DC, resolved this with fixed, 9MB piecewise MD4, but this segment size scaled poorly, ensured that fixing corruption demanded at least 9MB of retransmission, and used the weak and soon-broken MD4. Gnutella, though, had found an elegant, scalable solution in TTH.

This Tiger Tree hash, which I thus copied from Gnutella, scales to both large and small files while depending on what was at the time a secure-looking Tiger hash function. It smoothly, adaptively sizes a hash tree while retaining interoperability between all such sizes of files files on a hub. By 2003, I had released BCDC++ which used TTH. However, the initial version of hash trees implemented by Gnutella and DC used the same hash primitive for leaf and internal tree nodes. This left it open to collisions, fixed by using different leaf and internal hash primitives. Both Gnutella and DC quickly adopted this fix and DC has followed this second version of THEX to specify TTH for the last decade.

Though it has served DC well, TTH might soon need a replacement. The Tiger hash primitive underlying it by now lists as broken due to a combination of a practical 1-bit pseudocollision attack on all rounds, a similarly feasible full collision on all but 5 of its 24 rounds, and full, albeit theoretical, 24-round pre-images (“Advanced Meet-in-the-Middle Preimage Attacks”, 2010, Guo et al). If one can collide or find preimages of Tiger, one can also trivially collide or find preimages of TTH. We are therefore investigating alternative cryptographic hash primitives to which we might transition as Tiger looks increasingly insecure and collision-prone, focusing on SHA-2 and SHA-3.

Filed under ADC, Developers, Documentation, General, NMDC

DC++ 0.810

February 5, 2013 by Leave a comment

We’re happy to annouce that a new testing release of DC++ is available from the official download page. Version 0.810 contains numerous fixes and small improvements over the last release but this isn’t going to be a large post with lots of explanations in this case; the changelog items mostly speak for themselves.

I’d like to highlight three improvements that might need more attention:

  • The Plugin API has been evolved further but it’s still not reached the stage where the plugin binaries are possible to distibute with the client. They’re still in a testing phase but binaries available from now to those who interested on testing. As long as the the final plugin distribution format is beeing implemented the nightly plugin binary builds are available in a simple .dll format at the Plugins archive of the DCBase Builds server.
    At the time of writing two fully functional and a test plugin are available. The LUA 5.2.1 client side Script Plugin is able to run all BCDC++ client side scripts. The Dev Plugin offers functions like monitoring the protocol talk, follow search requests (former Search Spy function) and more. Make sure you consult the readme file before downloading any revision of any plugins.
    For those who interested in creating plugins the source code,  SDK and Doxygen documentation is still available in the DCPP Plugins SDK repositories (C and C++).
  • Fixing of two the two following long standing problems required to change the hash data file format: the case insensitive share format that caused problems on non-Windows clients based on the dcpp library and a bug with merging shares  containing same filenames/folders resulted an inconsistent behaviour.
    Because of these fixes if you upgrade from an eralier release the first start of the client might require a bit more time than usual. An automatic share format conversion will make you wait a bit more, depending on how large your share is.
    The bad news is that the automatic conversion is possible only on Windows Vista and later. Due to lack of extended file handling capabilities of the OS, the entire share needs to be rehashed for XP users.
  • The only remarkable and very useful GUI improvement this time is the ability to copy data from every list view to the clipboard. It is possible through the lists’ context menus, using a new ‘Copy’ menu item. You are able to copy the content of one column or the data of all columns in one go.
    There’s also a special new menu item  (called “Copy user information”) for lists containing users; it’s made to easily gather all the available info (not just what’s displayed) about one user or multiple selected users copied to the clipboard.

Things looking good so far; this means that DC++ 0.810 should make stable within days. Upgrade is recommended as always.

Filed under Clients, General, Plugins, Release

ADC 1.0.2 released

February 3, 2013 by Leave a comment

A new version of the base ADC protocol is now released, version 1.0.2.

The document may look slightly different, especially with the addition of commands in the table of contents. The document itself (its content) is not that much modified (except for state management, see below).

An important part of the document is a new addition, a terminology section where difficult words or phrases are specified. This list is obviously meant to be much more than mere four items but it’s at least a start.

The STA previously didn’t specify who had the responsibility for action when a STA is sent with the severity Fatal (2). This has always been the originator of the message, and this is now explicit.

The state management is re-worded and restructured. All information about state has now been moved to its own section, allowing an implementator a quick and comprehensive overview on the requirements for the state management. Previously, the state management was sprinkled all across the document, making it difficult for a person to properly implement a state machine in their software. This has meant that state management information is now removed from each command (only thing remaining is an explicit note about in which state each command is used). Certain information is also clarified, such as what to call the parties in a client to client connection (“client party” and “server party”) and state transitions.

Version 1.0.1 of ADC was also ambiguous in state management when it came to one important part: who shall send the first INF in a client to client connection. This is important because it has the ramification that it makes multi-share difficult. The current specification is now not ambiguous, and makes the following stance: the first party to send the INF is the connecting party (“client party”). No known implementation suffer from this explicit note, as all manage this scenario just fine. Basically, this change means that multiple shares (per hub) may not be too far off.

The new version also brings in a new time where we can safely and appropriately update the base document. There was an announcement period when the document was going to be released which meant that developers have had time to adjust their software and give feedback in a timely manner.

Filed under ADC, Documentation

DCDev archives published

January 3, 2013 by Leave a comment

I previously requested the DCDev archives, a repository of posts from DC developers. I was able to acquire the repository and it is now posted on DCBase.

There’s a lot of stuff in the posts, especially the initial parts with ADC. Enjoy.

Don’t forget that you can make topic suggestions for blog posts in our “Blog Topic Suggestion Box!”

Filed under General

Request for video about attacks spawned from DC

January 3, 2013 by Leave a comment

In a previous post, the security on DC was discussed as well as an attack on hublist.org, the then largest hublist for DC. This attack, as well as a generic view on attack on the web, was covered on a Finnish program (translated to Enligh as “MOT: Invisible plane hijacking”). This was broadcasted a few years ago (see the link) but wasn’t uploaded to their site.

My intention was to get this video and upload it to the Youtube page so everyone can view it. I have aquired a copy of it from a video recording, but I cannot simply (re)broadcast it because of obvious legal reasons.

This is a request for anyone who are able to come in contact with YLE.fi (the broadcasting company) and are able to persuade them to either upload the video or allow us to upload it.

I have previously made a request to YLE but was denied due to the fact that the video contains a trailer from the movie Die Hard. YLE claim they do not have the necessary resources to edit this film. (I was denied the ability to do the edits myself.)

YLE sales are in charge of the video and I can provide details of the person(s) I’ve been in contact with, if you think you have a better chance.

Filed under General

DC++ 0.802

October 21, 2012 by Leave a comment

A new stability and performance update is available for DC++. The new version brings a nice global performance improvement and a further boost focused on a couple of other areas. It also fixes a severe stability problem as well.

This new version introduces a different global thread mechanism that should make the client faster and able to utilize some features of modern CPUs. The cost is that DC++ requires a P6 (Pentium Pro / AMD K6) or newer processor from now to run. As most of the currently supported operating systems require even newer CPUs, this should not be a problem for the vast majority.

DC++ 0.802 should considerably improve the client’s performance under heavy load as well (e.g. beeing logged on to many hubs with large usercount) and decrease the resource consuption in these cases.

You might already learned that file list loading operations are improved in version 0.801. The good news is that due to further optimization and a caching algorithm you should experience another boost in responsiveness when you open large file lists with DC++ 0.802.

This release also fixes a random crash problem that can produce weird and nonsensical crash reports. Though 0.802 should go stable within a few days, we’d like to ask everyone, users and betatesters, to report crash logs made by version 0.802 only from now.

The detailed changelog is in the repository as usual.

Filed under Clients, General, Release

DC++ 0.801

October 20, 2012 by Leave a comment

The first stable version of the 0.8x series of DC++ is available to download for a while. We already posted about the largest improvement of the new release that marks a new era for the DC network, a fixed critical security vulnerability as well as about Windows 8 compatibility. Now it’s time to go through the rest of the  improvements and developments that came with DC++ 0.801 by picking the remaining interesting items from the changelog with occasional comments where needed.

GUI improvements

  • Revamp favorite hub settings

Various global  settings are now possible to be overridden per favorite hub. Opening the fav hub properties dialog they should be self-explanatory.

  • Tweak help tooltips in the settings dialog

Tooltips should not cover any important parts of the dialog anymore.

  • Make the menu bar hideable
  • Replace the slot up-down control by a context menu

Easier to change the number of upload slots while it causes less strain to hubs and other users.

  • Allow Magnet links to be pasted in the “Quick connect” box

Since the torrents network picked up magnet links they become more and more popular. As there is a higher chance  to find magnet links outside the DC network here’s the place where you can directly import them.

  • Remember list sorting & splitter positions on various places

Client functionality changes

  • Rise the minislot size to 512 KiB

DC++ has limited number of upload transfers at the same time and this is regulated by upload slots (strange that after 10+ years of exitstence of the DC network some people still haven’t realized this: “No slots available” is still often searched within this blog). There are different type of slots and they were already explained in this post. In DC++ 0.800 the default mini slot size rose to 512KB and this will allow users to get small files much sooner than before. The previous default value was chosen long before when Internet connection speeds were much lower. The change is made because nowadays extra transfers of files smaller than 512KiB should not be a burden for the majority of connections. The setting remains configurable (though it’s possible to set a higher value only).

  • Upload queue postition (QP) support

QP or ‘Upload Queue Notification’ is a client side mechanism that creates a queue of connecting users on a “first come, first serve” basis when your slots are full. QP works like this (say you only have 1 slot and it is free), users “a”, “b” and “c” connect to you in ascending order user a can begin his/her file transfer immediately. Users b and c are put in a queue 0 and 1 respectively, as soon as user “a” is finished user “b” can start and user “c” then moves up the queue to position 0. Basically all this means to you (the user) is it creates a fair system where everybody gets a slot in the order that they connected so you don’t end up getting bumped whenever a slot becomes free to someone else who can connect faster ( if you go offline you will lose your Queue Position). The actual queue position is indicated in the Status column of the Connections part of the Transfers window. Please note that QP is fully implemented for ADC hubs only.

Bug fixes, stability, performance

  • Reduce the resource consumption some when upload slots are full
  • Don’t choke on hub addresses with spaces

This problem was already detailed here.

  • Fix a mixup between IPs and hostnames leading to wrong search results

Is a problem in NMDC hubs only. Sometimes the search result contains the host name instead of the IP of the responder’s hub address rendering the search result unprocessable by the requesting client. This bug is introduced in DC++ 0.790 thus hits all 0.79x versions.

  • Fix glitches with the file list loader; Eliminate GUI freezes when opening large file lists

Various problems fixed while loading large filelists as well as the speed and responsibility of the client is greatly improved while file lists are beeing opened. The next version of DC++ will improve even further in this area.

  • Fix buttons not available in user matching settings panel in some cases
  • Improve performance when selecting lots of lines in lists
  • Greatly improve user command removal time when closing a hub
  • Grant extra slot hangs connection in ADC hubs

Was a really old problem, and is fixed at last.

  • Fix NAT-PMP renewal

NAT-PNP is still a gray area because not too many router devices feature this protocol yet. Anyone with a NAT-PMP capable router is more than welcomed to help us testing and to polish the support of this nice and simple port mapping protocol in DC++.

  • Fix GeoIP & OpenSSL problems with wide character paths

This bug could bother you if the path of your DC++ Certificates folder or the DC++ settings folder itself contains non-latin characters. Users from e.g. Russia, China etc… having non-latin characters in their Windows username could fall into this category. The problem blocked operations associated with these folders, for example the automatic upgrade of GeoIP information.

The list of fixes are large and DC++ 0.801 is already made stable so it is a highly recommended update for everyone. For the complete list of changes please refer to the changelog.

Filed under Clients, General, Release

The evolution of connection setup in DC++

October 20, 2012 by Leave a comment

You might already know that there was yet another connection setup change in the latest release of DC++. Instead of writing a few sentences only about the current changes I thought I’d rather show you the best part of the evolution of DC++ regarding to the connection setup.

In the beginning and for a very long time the connection setup was the most hated part of DC++ for many people. But with the success of  the Automatic connectivity setup implementation setting up the connection manually has become an advanced feature in DC++; in the majority of cases an average user should not modify or even check these settings.

The connection setup layout, labeling and the number of active mode options are often changed over the history of DC++, especially in last couple of years. The four available active mode options that has been part of the setup page during the recent years are debuted back in DC++ 0.68. That time they were expanded compared to the eariler releases and the options has become more descriptive for the sake of better understanding.  However, the need of manual setup of active mode  still remained a problematic task for many users for many more years, until the first incarnation of automatic connection setup function has introduced in DC++ 0.780.

At the time of writing the last change in the connection setup GUI is that with DC++ 0.800,  the number of active mode options shrinked from four to three.  As two of the existing manual active mode setup options actually had identical function now they are simply merged: you should use the same option for direct connection and manually port forwarded active modes from now.

Here’s some history in pictures showing how the setup functions and layout has been changing over the years. More of them can be found by Google.

This is something from the ancient times. There’s even no separate Connections page yet.

This one’s  from around cca. version 0.670. Settings moved to a new page but no change in the layout. In active mode DC++ automatically tried UPnP port mapping if the ports input boxes were empty and automatically went for passive mode in case of a mapping failure. UPnP implementations were pretty unreliable that time (both in DC++ and in the routers) so there were many non-working and passive setups. But most people still loved it because it often gave some sort of working connectivity without playing with the settings.

The layout we had for many years. Introduced in 0.68 and people really hated it because #1 old settings were not converted on upgrade #2 in most cases it isn’t worked without touching the settings and it threw various error messages #3 the options and error messages were confusing (eg. most people couldn’t quite understood what “Firewall” means here). This layout had lasted until DC++ 0.75.

DC762settings

Here’s the first attempt to have more meaningful labels for the options. It is from the era when DC++ rapidly started to become more user friendly (version 0.76x). Still no automatic setup yet.

First incarnation of automatic setup in version 0.78x.  Manual settings are still on the same page as the automatic ones and they getting greyed out when autoconfig is selected. The first time when DC++ has some indication of what happens during the connection setup so problems were lot easier to solve using the log. By this time DC++ already used MiniUPnP for port mapping which resulted a lot more successful port mapping attempts, really. This release was an overall success throughout the community.

 

This is DC++ 0.79x. Manual config moved to a separate page and you can copy and edit autodetected settings in the manual page. This version introduces NAT-PMP as an alternative way for automatic port mapping. At last, bind address setting is placed at where it should be.

 

…and finally the current relabeled and optimized manual options in 0.800. Mind that while they’re lot more versatile, due to the last optimization the options are also more simple and a bit similar to the very old layout.

Will connection setup evolve more with time in DC++? That I don’t know but I am sure that DC++ already in par with other recent P2P applications in this regard.

Filed under Clients, General

The road ahead: Post summary

October 19, 2012 by Leave a comment

This series I called “the road ahead” is simply my look into how we can improve and continue to develop the DC community and system. The posts should be readable as they are but can of course be read in conjunction with each other. There’s no priority in the suggestions I make. Instead, everyone should look at the posts and think about what small part they can do to help accomplish some of the suggestions (or if you have some of your own).

The posts:

  • Security and Integrity – About security and integrity in DC and how it affects us
  • Protocols – What the protocols should strive for and the tools the protocol community should provide
  • Competition – The different challenges that we face in Direct Connect in the battle of users
  • Software – What improvements we can make to software across the board
  • Widening the base – How the different information outlets we have can increase DC knowledge in the world
  • Infrastructure – How the infrastructure in DC can improve

Filed under General

The road ahead: Infrastructure

October 19, 2012 by Leave a comment

Direct Connect rests upon three parts: clients, hubs and hub lists. If there are no clients available then the community becomes stagnant and the appeal for new users diminishes.  Hubs must be available, else there’s no way for clients to connect to each other. The hubs provide the very community we have. The hublists provide clients with a sense of direction where there are other hubs with users.

These three parts are, in my mind, equally important and it is imperative that we have them in our infrastructure.

Direct Connect has had problems when hublists go down or becomes outdated etc, so the infrastructure we have should manage these types of problems.

The Direct Connect community is concentrated around the ability to provide a straight forward file sharing service all the while having a talk and discussion forum. These two parts are why DC is so great: you can have a discussion while sharing content that you and your peers like.

The infrastructure of DC should allow users to interact while they are not necessarily using a normal client. A website can serve as a client where you simply have a tab for chat and one for downloading etc.

The infrastructure should give users the ability to browse any type of software or discussion topic around DC, and this is the intent with DCBase.org. The idea is to create a central source for DC content. If we can gather information about each client in one place, users don’t need to go to different sites that have different appearances. users can go to a single site and if they choose so, they can continue on to the main page of the mentioned software or item of interest. The central source needn’t own each client or have the source code of each client, it simply needs to be able to refer to them.

Basing content around one place will also help making information unambiguous and not redundant. This infrastructure idea should allow developers control over their system while the face of DC can be unanimous.

An interesting aspect may be to create, say, a real non-profit organization. The organization could be recognized by a government, allowing the validity of DC to increase and potentially draw some attention from new users. The organization can serve as an ‘umbrella’ for donations, for example by distributing its donations to developers and their own infrastructure. It could also open up the possibility of receiving government funds to stabilize the DC infrastructure.

The infrastructure can provide websites, build and download repositories. It can allow specialized hubs dedicated to support and development.

The road ahead

The future is to try and merge many sources of information that needn’t be separated. For instance, the NMDC and ADC project can simply be a general “DC protocol project”, with minor branching. Whenever we can join resources, it will mean that there’s less time managing the multitude of sources and more time doing what we want — the further development of DC. If we can merge different functionality of software or if we can provide a clear interface for those interested in DC content, all the better.

Filed under General

← Older posts

Newer posts →

Design a site like this with WordPress.com
Get started