OaShare

Inspiration

Ciena wanted a TCP-based file sharing system. So we decided to build the best one we could!

What it does

• Uploads files: Splits files into chunks of 512 bytes (+ headers) and spreads them out across a local network!
• Scans possible peers: No input necessary! Automatically find peers in your local network.
• Download files: Need those files back? Ask the peers in your network and get the file back piece by piece!
• Integrity verification: Can't have files running around upside down and backwards!
• Retries and some fault tolerance: Network not cooperating? Just try again!

How we built it

• Python
• SQLite
• asyncio

Originally, this was supposed to be a socket and threading implementation, but then we realized that async was technically fewer lines of code . . . and less worrying about threads and locking.

Challenges we ran into

• Windows Firewalls existing (which frequently blocked access). We eventually used a Docker setup with 4 containers for nodes, which worked much more smoothly.
• How do you get a file onto another computer, let alone split it up into chunks and then download it later? This drew some inspiration from TCP itself, including chunk ordering and checksums.
• How do you know who's on your network? And which computers yours can talk to?
• Async is pain -- Figuring out async workers and task queues and what coroutines to use where took some fiddling, and some questions to Perplexity.

Accomplishments that we're proud of

• Pure TCP Protocol: No HTTP, no JSON, no gRPC, all TCP!
• 4 Node Docker Compose setup: An easily scalable* testbed for stress testing!
• Async Socket Server: Pure Python Async I/O (minus the database), allowing for better capacity and easier development!
• Functional up- and downloads: Splitting up a file and piecing it back together doesn't sound like a lot . . . until you realize you're two days in and you still haven't gotten a file across.

*once we figure out how to schedule tasks and get better than 75% successful download rates.

What we learned

• TCP protocols are hard. Researching existing protocols was helpful in developing our own.
• Async is super viable for socket programming, maybe even easier than regular threads!
• Databases are ironically better for sharing file chunks than actual filesystems.
• We learned how to get a file from point A to points B, C, and Z!
• We learned how to get your network neighbours from Python regardless of what network you're on!

What's next for OaShare

• Better fault tolerance and robustness
• End to End Encryption
• Maybe a rewrite in Rust?
• Proper UI

Built With

• docker
• python
• sqlite