Skip navigation.
Home

Development


PeerStreamer is composed of several software components. To get a build environment with the source code of all the parts, follow the two easy steps in the developers' Quick Start guide below. This will download all the necessary components, compile the libraries, then the streamer.

Quick Start

Prerequisites

Linux

Install the following development tools:

apt-get install git gcc g++ subversion libtool autoconf yasm texinfo libx11-dev libxext-dev curl

Note: on old systems you need git-core instead of git.
Note2: you can use wget instead of curl.

You might also need some of the following packages (to be confirmed, seems to solve the "SDL_OpenAudio: Text has zero width" issue):

apt-get install libsdl-sound1.2 libsdl-sound1.2-dev libavcodec52 libavdevice52 libavformat52
apt-get install libxext-dev

Windows

Windows buids are also prepared on Linux using the MinGW build environment.
To compile the windows version, install the following:

apt-get install gcc-mingw32

Note that the older mingw32 package has some issues with headers, and it might fail in the ffmpeg compilation part.

You can use Wine to test the compiled Windows version of the PeerStreamer package directly in the compile environment.

Mac OS X

You will need the Xcode Developer Tools package (from Apple), git, and yasm (for x264) to build our code. There are many
ways of getting them, here is one:

1, install Xcode Developer Tools from your resource DVD or download it from Apple

2, download and install git from here

3, download and install yasm

curl http://www.tortall.net/projects/yasm/releases/yasm-1.1.0.tar.gz > yasm-1.1.0.tar.gz
tar -xvf yasm-1.1.0.tar.gz
cd yasm-1.1.0/
./configure ; make
sudo make install

Building on Linux (both 32-bit and 64-bit) and OS X

1, Clone the Streamer-build repository

git clone git://halo.disi.unitn.it/cskiraly/Streamers-build.git && cd Streamers-build

2, Make PeerStreamer (use -j for a faster build)

make -j 4

Building for Windows

We build Win versions on Linux, using the MinGW cross-compiler. This builds a 32-bit binary, which runs fine both on 32-bit and on 64-bit Windows versions (XP, Vista, 7). Follow these steps:

1, Clone the Streamer-build repository

git clone git://halo.disi.unitn.it/cskiraly/Streamers-build.git && cd Streamers-build

2, tell your shell that you are cross-compiling by including the setenv-mingw file in the environment

source setenv-mingw

3, Make PeerStreamer (use -j for a faster build)

make -j 4

Note: If you use the same build directory both with the cross-compiler and without it, you'll end up having a mess. Do a make distclean or restart from scratch.

Detailed Instructions

Build Environment

We use git for version control. It is used both to provide a consistent build environment, as well as to maintain the source code of various software components developed or used. The code of the build environment is available here.

Getting the Build Environment

You can check it out directly with the following command:

git clone git://halo.disi.unitn.it/cskiraly/Streamers-build.git && cd Streamers-build 

Getting all the Code before compilation (optional)

The build environment can downloads all of our code, as well as external dependencies, in one shot. The goal is to synchronize versions across all these components, so that no problems due to version mismatches arise. Note that this is also automatically triggered by the build process so you can skip this step. Just run

make prepare

to get all the code.

Changing versions (optional)

You can easily change among tagged versions (or branches) of the code through git commands. Use

gitk --all

to visualize the history and various branches of the build environment and the applications.
In general, we keep at leas two branches up-to-date: the stable master branch, and the next branch we are currently experimenting with. Other topic branches might also be available.

You can change to e.g. the next branch with the following commands

git checkout next
git submodule update

Note: in older versions of git, you might have to replace the first command with "git checkout -b next origin/next".

Compiling the code

Just run

make -j 4

(use -j for a faster build).

If all goes well, you will find the binary in the Streamer folder.

Updating the code

In most cases, if you have not modified the code, a simple

make update
make -j 4

does the trick.

If you have modified the code, you can either handle conflicts manually, or use

make forceupdate
make -j 4

Note that this will put aside your changes running a "git stash" in the main project and in each submodule.

Software Components Explained

Streamer application

The Streamer application is our main application. Besides being the code that implements the streaming protocol and the application logic, it is also the code that binds together all the components into one executable. Before working with/on the Streamer's code, you have to make some decisions. You have to chose an incarnation, provide some compile-time configuration, and finally, do some run-time configuration. Don't panic, defaults provide a good start. Change them only if you understand the effects!

Streamer incarnations

The Streamer code has several variants, implementing various streaming protocols or targeting various audiences. We call these "incarnations". You can find:


  • End-user oriented incarnations (only WineStreamer as of now): these focus on performance, features, and ease of installation/use. We might e.g. change from push-based protocols to pull-based ones from one version to another if it fits better our performance goals. The code might also contain some dirty hacks to make things work.

  • Research oriented incarnations (DumbStreamer, RockStreamer, OfferStreamer): these focus on one specific streaming protocol or algorithm, keeping other features a bare minimum. These are "no-frills" versions! The goal of these is to study an algorithm without interfering with other algorithms, heuristics.

Incarnations are available from the git repository as branches. You can browse the code of these here

Streamer compile-time configuration

Since we have several alternative implementations for many of the components, the Streamer is highly configurable at compile time. You can decide the type of the I/O used, the type of the messaging layer used, and many other parameters of the executable at compile time. Until you want to explicitly change one of these, defaults will be fine. Defaults change a bit from incarnation to incarnation. End-user oriented incarnations (WineStreamer) tend to include all the libraries. As a result, you get more complex code, less control of what is happening, but more functionality. Development and research oriented are configured by default to use basic versions of the libraries.

Streamer run-time configuration

The streamer has some incorporated help (with examples). Run it with the -h option to get it.

GRAPES library

GRAPES (Generic Resource-Aware P2P Environment for Streaming) is the core library behind PeerStreamer. It provides data structures and algorithms for chunk scheduling, topology management, and media-aware input/output. It also implements signaling and chunk transmission network primitives.

One of our principal design concepts was to separate these libraries from PeerStreamer, providing a generic library that could be used in other applications as well. See the GRAPES page for more information about the library.

Input/Output

the PeerStreamer framework currently includes two I/O modules. The choice has to be made a compile time passing the "io" parameter to make.

  • "io=grapes": in this case, the I/O module of the GRAPES library is being used. This module provides chunkization at the input and de-chunkization at the output. It also handles various media formats (using libavformat) and it can transport media almost independent of the codec. It does not support transcoding (but transcoding can easily be achieved at the source by ffmpeg or any other similar program). We mainly use this module in automated tests, measurements and experiments.
  • "io=chunkstream": this I/O module links the streamer to the external Chunker and Player applications. On the source side, the Chunker supports transcoding to various codecs (we usually use H.264 + mp3). On the peer side, the Player is an individual media player application that also supports channel switching and various other features. We use this I/O module in the end-user
    version of the application.

Messaging

PeerStreamer can be configured with one of the following two messaging libraries:

  • The UDP based messaging module included in GRAPES: this simple module provides arbitrary message size, and a simple code that is easy to start with. It is ideal while testing other parts of the code. Note that its current version is not recommended for large-scale deployments, as it misses NAT support.
  • The Messaging Library: This UDP based messaging library, provides several advanced features, including: arbitrary message size; STUN support; NAT-traversal; NAT-local shortcuts; message rate shaping; packet level retransmission; packet and message monitoring hooks. Its use is recommended for deployment versions.

ALTO integration

Our topology management module is integrated with our open source ALTO client library (which you can also find in our repositories here)

Monitoring

The streamer is integrated with the MONL Monitoring Library. This library provides us with detailed distributed low-level network measurements, hopefully making our algorithms smarter.

Repository publishing

The streamer also incorporates an optional reporting framework, designed to support scientific experimentation. Peers can report performance statistics (turned off by default) to the ModularRepository. Data is post-processed and can easily be visualized with the SwarmVisualizer. Both of these applications have been developed in the framework of the NAPA-WINE project, they are both open source.

Testing the code

We continuously develop and use test environments together with PeerStreamer. We test it on one machine; with virtual machines; on PlanetLab; as well as in real use.

Testing on a single PC

You can test the Streamer or your modifications to it on a single PC, running a source and a number of peers (watch out not to overload your CPU ;) ).
We have developed a script that does it all, from implementing churn to gathering debug information!

First of all, you will need a Streamer variant that uses the GRAPES I/O (by default the streamer is compiled with Chunkstream I/O, which is not working with the test). Prepare the version that uses GRAPES I/O with:

make ml

Next, clone the Streamers-test repository:

git clone git://halo.disi.unitn.it/cskiraly/Streamers-test.git

You are almost ready to run tests with a source and several peers, you just need to have some video file! Our main test script is Streamers-test/test.sh. You can start a test by e.g. running the following command:

Streamers-test/test.sh -e Streamers/streamer-ml-monl-grapes-static -v ~/video/big_buck_bunny_480p_1Mbps_mpeg4_mp2.ts

You will see the stderr output of the source in the terminal. The output of the peer will be recorded in the stderr.6667 file.
Note: test.sh uses the -v option to define the input file! You can find more documentation here, in the man page of the script, or by looking inside the script itself!
Note: you can download an example video file for the tests from here

Functional testing

You can run various functional tests using test.sh. Some examples follow.

Run a source and a peer
Streamers-test/test.sh -e Streamers/streamer-ml-monl-grapes-static -v ~/video/big_buck_bunny_480p_1Mbps_mpeg4_mp2.ts
Run a source and a peer, loop the video file infinitely
Streamers-test/test.sh -e Streamers/streamer-ml-monl-grapes-static -v ~/video/big_buck_bunny_480p_1Mbps_mpeg4_mp2.ts,loop=1
Run a source and 20 peers
Streamers-test/test.sh -e Streamers/streamer-ml-monl-grapes-static -v ~/video/big_buck_bunny_480p_1Mbps_mpeg4_mp2.ts -N 20
Run a source, 10 stable peers, and 10 "churning" peers
Streamers-test/test.sh -e Streamers/streamer-ml-monl-grapes-static -v ~/video/big_buck_bunny_480p_1Mbps_mpeg4_mp2.ts \
 -N 10 -C 10 -t 15 -T 30
Run a source, 10 peers in the background, 3 peers in separate X terminals
Streamers-test/test.sh -e Streamers/streamer-ml-monl-grapes-static -v ~/video/big_buck_bunny_480p_1Mbps_mpeg4_mp2.ts \
 -N 10 -X 3
Run a source and a peer with video output
Streamers-test/test.sh -e Streamers/streamer-ml-monl-grapes-static -v ~/video/big_buck_bunny_480p_1Mbps_mpeg4_mp2.ts \
 -N 0 -O 1 -o "THIRDPARTY-LIBS/ffmpeg/ffplay -"

You can also use other player like e.g. VLC.

Streamers-test/test.sh -e Streamers/streamer-ml-monl-grapes-static -v ~/video/big_buck_bunny_480p_1Mbps_mpeg4_mp2.ts \
 -N 0 -O 1 -o "cvlc -"
Run a source, check the video output of 3 peers, and add 10 more peers in the background
Streamers-test/test.sh -e Streamers/streamer-ml-monl-grapes-static -v ~/video/big_buck_bunny_480p_1Mbps_mpeg4_mp2.ts \
 -N 10 -O 3 -o "THIRDPARTY-LIBS/ffmpeg/ffplay -"

Testing for memleaks and other nasty things

test.sh is integrated with Valgrind, so you can easily test your modifications for memleaks and other memory issues. Follow the instructions below to get test environment.

First of all, install Valgrind:

sudo apt-get install valgrind

You need a dynamically linked version of the streamer, since Valgrind is not working with static binaries:

STATIC=0 make ml

You can run your tests. -V sets the number of peers that use Valgrind. Note that peers running under Valgrind consume much more CPU cycles, so you are better off with one peer running under Valgrind and some other peers without it (you might need more peers to e.g. test topology related code).

LD_LIBRARY_PATH=THIRDPARTY-LIBS/NAPA-BASELIBS/3RDPARTY-LIBS/libevent/lib Streamers-test/test.sh \
 -e Streamers/streamer-ml-monl-grapes -v ~/video/big_buck_bunny_480p_1Mbps_mpeg4_mp2.ts -N 10 -V 1

Note the differences compared to the previous examples:

  • since you use a dynamically linked binary, you need to specify libevent library path
  • -e Streamers/streamer-ml-monl-grapes: we use the dynamically linked version of the binary
  • -V 1: start a peer under Valgrind
  • -N 10: we run some more peers to activate topology code

After some seconds, you can kill the process with CTRL-C. Check the recorded stderr.* files for Valgrind output.

Testing the version with chunkstream I/O

This is a preliminary description of how to test the version with chunkstream I/O, i.e. the version that has a separate chunker application and a dedicated player application.

First, make this version

make

This will create a folder called PeerStreamer-$VERSION. Change to this deriectory, then run the source

 ./source.sh -f ~/video/big_buck_bunny_480p_1Mbps_mpeg4_mp2_scth.ts -V libx264 -A mp2 -s "-I lo"

Note that we chose to encode video in h.264, using the libx264 library, and we keel audio as mp2 for simplicity. We run the source on localhost to simplify instructions.

Before running the player, we need a channel file that describes the channel we've just started. This is not yet autogenerated, so we have to do this by hand. Create the file channels-local.conf and add the following

Channel TEST-CHANNEL-01
{
	LaunchString = "-I lo -i 127.0.0.1 -p 6666"
	AudioCodec = mp3
	AudioChannels = 2
	SampleRate = 48000
	VideoCodec = h264
	Width = 854
	Height = 480
	Ratio = 1.7777
	Bitrate = 1000000
}

Now, it is time to start the player

./player.sh -C channels-local.conf

You should see the GUI starting up, tuning on the channel in a breeze.

Contributing to development

Don't worry, you can contribute to the development of PeerStreamer in many ways, not just by writing code. You can report issues, discuss features or feature requests on our mailing list, start your own channels and provide feedback, etc.

If you want to contribute to code development, you can clone our repositories directly on GitHub.