Add detailed outline and references for decentralized music streaming thesis
This commit is contained in:
20
diagrams-plan.md
Normal file
20
diagrams-plan.md
Normal file
@@ -0,0 +1,20 @@
|
||||
# Planned Diagrams for System Design (Chapter 3)
|
||||
|
||||
| Section | Diagram Type | Description |
|
||||
|---|---|---|
|
||||
| 3.2 Representing Music in AP | Code listing | JSON-LD examples showing the custom vocabulary extension (Track, Album as OrderedCollection, BLAKE3 hash embedding) |
|
||||
| 3.3 Federation Model | Sequence diagram | Instance A uploads track → federates to instance B via Create activity → instance B's listener discovers it |
|
||||
| 3.4 Content Distribution | Flow diagram | Upload → instance seeds via iroh → desktop client requests byte range → starts playback. Show NAT traversal (hole punch / relay fallback) |
|
||||
| 3.5 Architecture | Crate dependency graph | Hexagonal layers: domain, application, adapters. Same style as thoughts/movies-diary Mermaid diagrams |
|
||||
| 3.5 Architecture | Component diagram | High-level: backend server, iroh node, desktop client, PostgreSQL, NATS. Show how they connect |
|
||||
|
||||
## Rendering options
|
||||
|
||||
- **Mermaid → SVG/PNG**: Export with `mmdc` CLI, include as images in Typst
|
||||
- **fletcher**: Typst package for in-document vector diagrams (nicer but more work)
|
||||
- **draw.io**: Manual option if the above don't fit
|
||||
|
||||
## Reference material
|
||||
|
||||
- `/mnt/drive/dev/thoughts/ARCHITECTURE.md` — Mermaid crate dependency graph + domain ports class diagram
|
||||
- `/mnt/drive/dev/movies-diary/architecture.mmd` — Mermaid hexagonal architecture (more detailed)
|
||||
32
outline.md
32
outline.md
@@ -20,9 +20,9 @@
|
||||
### 2.2 Peer-to-Peer Content Distribution
|
||||
- Overview of P2P network topologies
|
||||
- BitTorrent protocol
|
||||
- WebTorrent (BitTorrent over WebRTC)
|
||||
- IPFS and content-addressed storage
|
||||
- Comparison of approaches and rationale for chosen solution
|
||||
- iroh: QUIC-based content distribution with byte range requests
|
||||
- Comparison of approaches and rationale for choosing iroh
|
||||
|
||||
### 2.3 Decentralized Systems Architecture
|
||||
- Hexagonal architecture (Ports and Adapters)
|
||||
@@ -39,7 +39,7 @@
|
||||
- Extending ActivityStreams 2.0 with a custom JSON-LD namespace
|
||||
- Audio object design (metadata fields, duration, attribution)
|
||||
- Album as OrderedCollection
|
||||
- Embedding content addresses (magnet URI / CID) in AP objects
|
||||
- Embedding content addresses (BLAKE3 hash) in AP objects
|
||||
- Interoperability considerations with generic AP clients
|
||||
|
||||
### 3.3 Federation Model
|
||||
@@ -51,8 +51,10 @@
|
||||
### 3.4 Content Distribution Model
|
||||
- Replication policies: eager vs on-demand
|
||||
- Instance as a stable seeding node
|
||||
- HTTP streaming proxy for browser and mobile clients
|
||||
- Native desktop/TUI clients as full P2P peers
|
||||
- iroh-blobs for content-addressed audio transfer
|
||||
- Byte range requests for streaming and seeking
|
||||
- Native desktop clients as full P2P peers
|
||||
- NAT traversal via QUIC hole punching and relay fallback
|
||||
- Storage policy configuration
|
||||
|
||||
### 3.5 Architecture
|
||||
@@ -64,12 +66,11 @@
|
||||
|
||||
## 4. Implementation
|
||||
### 4.1 Technology Stack
|
||||
- Rust backend (Axum, SQLx, activitypub_federation crate)
|
||||
- Rust backend (Axum, SQLx, k-ap federation crate)
|
||||
- PostgreSQL
|
||||
- NATS for async event fan-out
|
||||
- Next.js web client
|
||||
- TUI client (ratatui + symphonia/rodio)
|
||||
- P2P library choice and rationale
|
||||
- iroh and iroh-blobs for P2P content distribution
|
||||
- Native desktop client (Rust GUI framework + symphonia/rodio for audio)
|
||||
|
||||
### 4.2 Backend
|
||||
- k-ap crate (reusable federation layer)
|
||||
@@ -77,14 +78,11 @@
|
||||
- P2P transport adapter
|
||||
- Federation worker
|
||||
|
||||
### 4.3 Web Client
|
||||
- Core user flows (upload, discover, follow, play)
|
||||
- HTTP streaming from instance proxy
|
||||
|
||||
### 4.4 TUI Client
|
||||
- ratatui interface
|
||||
- Audio playback pipeline
|
||||
- Direct P2P swarm participation
|
||||
### 4.3 Desktop Client
|
||||
- Rust GUI framework
|
||||
- Audio playback pipeline (symphonia/rodio)
|
||||
- Direct P2P participation via iroh
|
||||
- Byte range streaming and seeking
|
||||
|
||||
## 5. Evaluation
|
||||
### 5.1 Correctness
|
||||
|
||||
191
references.bib
Normal file
191
references.bib
Normal file
@@ -0,0 +1,191 @@
|
||||
% === Core Protocols ===
|
||||
|
||||
@techreport{w3c-activitypub,
|
||||
author = {Christine Lemmer-Webber and Jessica Tallon and Erin Shepherd and Amy Guy and Evan Prodromou},
|
||||
title = {{ActivityPub}},
|
||||
institution = {World Wide Web Consortium ({W3C})},
|
||||
type = {{W3C} Recommendation},
|
||||
year = {2018},
|
||||
month = jan,
|
||||
url = {https://www.w3.org/TR/2018/REC-activitypub-20180123/},
|
||||
}
|
||||
|
||||
@techreport{w3c-activitystreams,
|
||||
author = {James M. Snell and Evan Prodromou},
|
||||
title = {{Activity Streams 2.0}},
|
||||
institution = {World Wide Web Consortium ({W3C})},
|
||||
type = {{W3C} Recommendation},
|
||||
year = {2017},
|
||||
month = may,
|
||||
url = {https://www.w3.org/TR/2017/REC-activitystreams-core-20170523/},
|
||||
}
|
||||
|
||||
@techreport{rfc7033,
|
||||
author = {Paul E. Jones and Gonzalo Salgueiro and Michael B. Jones and Joseph Smarr},
|
||||
title = {{WebFinger}},
|
||||
institution = {Internet Engineering Task Force ({IETF})},
|
||||
type = {{RFC}},
|
||||
number = {7033},
|
||||
year = {2013},
|
||||
month = sep,
|
||||
doi = {10.17487/RFC7033},
|
||||
url = {https://www.rfc-editor.org/rfc/rfc7033},
|
||||
}
|
||||
|
||||
@techreport{w3c-jsonld11,
|
||||
author = {Gregg Kellogg and Pierre-Antoine Champin and Dave Longley},
|
||||
title = {{JSON-LD 1.1}: A {JSON}-based Serialization for Linked Data},
|
||||
institution = {World Wide Web Consortium ({W3C})},
|
||||
type = {{W3C} Recommendation},
|
||||
year = {2020},
|
||||
month = jul,
|
||||
url = {https://www.w3.org/TR/2020/REC-json-ld11-20200716/},
|
||||
}
|
||||
|
||||
% === Peer-to-Peer ===
|
||||
|
||||
@misc{cohen2003bittorrent,
|
||||
author = {Bram Cohen},
|
||||
title = {Incentives Build Robustness in {BitTorrent}},
|
||||
year = {2003},
|
||||
howpublished = {Workshop on Economics of Peer-to-Peer Systems},
|
||||
url = {https://www.bittorrent.org/beps/bep_0003.html},
|
||||
}
|
||||
|
||||
@misc{benet2014ipfs,
|
||||
author = {Juan Benet},
|
||||
title = {{IPFS} -- Content Addressed, Versioned, {P2P} File System},
|
||||
year = {2014},
|
||||
eprint = {1407.3561},
|
||||
archiveprefix = {arXiv},
|
||||
primaryclass = {cs.NI},
|
||||
url = {https://arxiv.org/abs/1407.3561},
|
||||
}
|
||||
|
||||
@inproceedings{maymounkov2002kademlia,
|
||||
author = {Petar Maymounkov and David Mazi\`{e}res},
|
||||
title = {Kademlia: A Peer-to-Peer Information System Based on the {XOR} Metric},
|
||||
booktitle = {Peer-to-Peer Systems: First International Workshop, {IPTPS} 2002},
|
||||
series = {Lecture Notes in Computer Science},
|
||||
volume = {2429},
|
||||
pages = {53--65},
|
||||
publisher = {Springer},
|
||||
year = {2002},
|
||||
doi = {10.1007/3-540-45748-8_5},
|
||||
}
|
||||
|
||||
@misc{webtorrent,
|
||||
author = {Feross Aboukhadijeh},
|
||||
title = {{WebTorrent} -- Streaming Torrent Client for the Web},
|
||||
year = {2013},
|
||||
howpublished = {\url{https://webtorrent.io}},
|
||||
url = {https://webtorrent.io},
|
||||
}
|
||||
|
||||
% === Fediverse Research ===
|
||||
|
||||
@inproceedings{raman2019mastodon,
|
||||
author = {Aravindh Raman and Sagar Joglekar and Emiliano De Cristofaro and Nishanth Sastry and Gareth Tyson},
|
||||
title = {Challenges in the Decentralised Web: The {Mastodon} Case},
|
||||
booktitle = {Proceedings of the Internet Measurement Conference ({IMC} '19)},
|
||||
year = {2019},
|
||||
publisher = {Association for Computing Machinery},
|
||||
doi = {10.1145/3355369.3355572},
|
||||
}
|
||||
|
||||
@article{lacava2021fediverse,
|
||||
author = {Lucio {La Cava} and Sergio Greco and Andrea Tagarelli},
|
||||
title = {Understanding the Growth of the {Fediverse} through the Lens of {Mastodon}},
|
||||
journal = {Applied Network Science},
|
||||
volume = {6},
|
||||
number = {1},
|
||||
pages = {64},
|
||||
year = {2021},
|
||||
doi = {10.1007/s41109-021-00392-5},
|
||||
}
|
||||
|
||||
@inproceedings{anaobi2023moderation,
|
||||
author = {Ishaku Hassan Anaobi and Aravindh Raman and Ignacio Castro and Haris Bin Zia and Dami Ibosiola and Gareth Tyson},
|
||||
title = {Will Admins Cope? Decentralized Moderation in the {Fediverse}},
|
||||
booktitle = {Proceedings of the {ACM} Web Conference 2023 ({WWW} '23)},
|
||||
year = {2023},
|
||||
publisher = {Association for Computing Machinery},
|
||||
doi = {10.1145/3543507.3583487},
|
||||
}
|
||||
|
||||
% === Architecture ===
|
||||
|
||||
@misc{cockburn2005hexagonal,
|
||||
author = {Alistair Cockburn},
|
||||
title = {Hexagonal Architecture},
|
||||
year = {2005},
|
||||
howpublished = {\url{https://alistair.cockburn.us/hexagonal-architecture/}},
|
||||
url = {https://alistair.cockburn.us/hexagonal-architecture/},
|
||||
}
|
||||
|
||||
@book{evans2003ddd,
|
||||
author = {Eric Evans},
|
||||
title = {Domain-Driven Design: Tackling Complexity in the Heart of Software},
|
||||
publisher = {Addison-Wesley},
|
||||
year = {2003},
|
||||
isbn = {978-0-321-12521-7},
|
||||
}
|
||||
|
||||
% === Distributed Systems ===
|
||||
|
||||
@book{tanenbaum2023distributed,
|
||||
author = {Maarten {Van Steen} and Andrew S. Tanenbaum},
|
||||
title = {Distributed Systems},
|
||||
edition = {4th},
|
||||
publisher = {distributed-systems.net},
|
||||
year = {2023},
|
||||
url = {https://www.distributed-systems.net/index.php/books/ds4/},
|
||||
}
|
||||
|
||||
@article{gilbert2002cap,
|
||||
author = {Seth Gilbert and Nancy Lynch},
|
||||
title = {Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services},
|
||||
journal = {ACM SIGACT News},
|
||||
volume = {33},
|
||||
number = {2},
|
||||
pages = {51--59},
|
||||
year = {2002},
|
||||
doi = {10.1145/564585.564601},
|
||||
}
|
||||
|
||||
@inproceedings{brewer2000cap,
|
||||
author = {Eric A. Brewer},
|
||||
title = {Towards Robust Distributed Systems},
|
||||
booktitle = {Proceedings of the 19th Annual {ACM} Symposium on Principles of Distributed Computing ({PODC} '00)},
|
||||
year = {2000},
|
||||
publisher = {Association for Computing Machinery},
|
||||
note = {Keynote address},
|
||||
}
|
||||
|
||||
% === Music Streaming Context ===
|
||||
|
||||
@misc{ifpi2024gmr,
|
||||
author = {{International Federation of the Phonographic Industry (IFPI)}},
|
||||
title = {{Global Music Report 2024}},
|
||||
year = {2024},
|
||||
howpublished = {\url{https://gmr.ifpi.org/}},
|
||||
url = {https://gmr.ifpi.org/},
|
||||
}
|
||||
|
||||
% === Existing Fediverse Platforms ===
|
||||
|
||||
@misc{funkwhale,
|
||||
author = {{Funkwhale Collective}},
|
||||
title = {{Funkwhale}: A Decentralized and Federated Audio Platform},
|
||||
year = {2018},
|
||||
howpublished = {\url{https://funkwhale.audio}},
|
||||
url = {https://funkwhale.audio},
|
||||
}
|
||||
|
||||
@misc{peertube,
|
||||
author = {{Framasoft}},
|
||||
title = {{PeerTube}: A Free and Federated Video Platform},
|
||||
year = {2018},
|
||||
howpublished = {\url{https://joinpeertube.org}},
|
||||
url = {https://joinpeertube.org},
|
||||
}
|
||||
30
thesis.typ
30
thesis.typ
@@ -39,7 +39,30 @@
|
||||
#pagebreak()
|
||||
|
||||
= Introduction
|
||||
#lorem(512)
|
||||
|
||||
Music streaming has become the dominant mode of recorded music consumption worldwide. Services such as Spotify, Apple Music, and YouTube Music collectively serve hundreds of millions of listeners, providing convenient access to vast catalogues of audio content. Despite their commercial success, these platforms share a set of structural properties that give rise to recurring concerns.
|
||||
|
||||
First, they are architecturally centralised: all audio content is stored on infrastructure controlled by a single corporate entity, creating single points of failure and placing the availability of the entire catalogue at the discretion of the platform operator. Second, they impose vendor lock-in on both listeners and artists: playlists, listening history, follower relationships, and catalogue presence are tied to a specific service and cannot be migrated.
|
||||
|
||||
An alternative architectural paradigm exists in the form of federated and decentralised systems. The Fediverse, a constellation of interoperable social platforms communicating via the W3C ActivityPub protocol @w3c-activitypub, has demonstrated that it is possible to build large-scale, user-facing applications without central ownership of data or infrastructure. Platforms such as Mastodon for microblogging, PeerTube for video, and Funkwhale for audio have shown that federation can provide practical alternatives to centralised services. However, the specific problem of music streaming in a federated context remains underexplored. Existing solutions such as Funkwhale store audio on individual server instances, meaning that content availability depends entirely on the continued operation of the hosting instance. If an instance goes offline, all content it exclusively hosts becomes unavailable.
|
||||
|
||||
Peer-to-peer content distribution offers a complementary approach. Protocols such as BitTorrent and systems such as IPFS distribute data across multiple participating nodes, so that content remains available as long as at least one node in the network holds a copy. Combining federation for social interactions and discovery with peer-to-peer transport for audio content could, in principle, yield a system where music remains available even when individual instances experience downtime.
|
||||
|
||||
This thesis investigates whether such a combination is feasible in practice. Specifically, it poses the following research question: _can a federated music streaming platform, built on ActivityPub for social interaction and peer-to-peer protocols for content distribution, provide acceptable content availability without relying on centralised storage?_
|
||||
|
||||
To answer this question, this work makes the following contributions:
|
||||
|
||||
+ *A reusable ActivityPub federation library.* The k-ap library, implemented in Rust, provides a domain-agnostic layer for ActivityPub federation, including inbox and outbox handling, WebFinger, NodeInfo, follower management, and activity broadcasting, that can be integrated into any application through a trait-based plugin architecture. The library has been validated in two prior applications: a microblogging platform and a movie diary service.
|
||||
|
||||
+ *An extension of the ActivityPub vocabulary for music.* The standard ActivityStreams 2.0 vocabulary lacks constructs for representing albums, tracks with duration and attribution metadata, and content-addressed audio references. This thesis defines a custom JSON-LD namespace that extends ActivityPub to model these concepts while preserving interoperability with generic ActivityPub clients.
|
||||
|
||||
+ *A hybrid federation and peer-to-peer architecture for music streaming.* The system design combines ActivityPub for content discovery and social interaction with a peer-to-peer transport layer for audio delivery. Server instances act as stable seeding nodes, while native desktop clients participate directly in the peer-to-peer swarm, reducing dependence on any single instance.
|
||||
|
||||
+ *A native desktop client.* In contrast to the browser-based clients typical of Fediverse applications, this work includes a native desktop client implemented in Rust, capable of direct peer-to-peer participation without the constraints of browser sandboxing.
|
||||
|
||||
+ *An empirical evaluation of content availability.* The system is evaluated through simulated network scenarios examining the effects of varying node counts, replication policies, content popularity distributions, and node churn on content availability.
|
||||
|
||||
The remainder of this thesis is structured as follows. Chapter 2 reviews the relevant background on ActivityPub, peer-to-peer content distribution, and decentralised systems architecture. Chapter 3 presents the system design, including the ActivityPub vocabulary extensions, federation model, content distribution model, and overall architecture. Chapter 4 describes the implementation, covering the technology stack, backend, and client applications. Chapter 5 evaluates the system against its stated requirements. Chapter 6 discusses known problems and limitations. Finally, Chapter 7 presents conclusions and directions for future work.
|
||||
|
||||
= Background and related work
|
||||
== The Fediverse and ActivityPub
|
||||
@@ -56,8 +79,7 @@
|
||||
= Implementation
|
||||
== Technology stack
|
||||
== Backend
|
||||
== Web client
|
||||
== TUI client
|
||||
== Desktop client
|
||||
|
||||
= Evaluation
|
||||
== Correctness
|
||||
@@ -68,4 +90,4 @@
|
||||
|
||||
= Conclusions and future work
|
||||
|
||||
= References
|
||||
#bibliography("references.bib", style: "ieee")
|
||||
Reference in New Issue
Block a user