README.md 13.2 KB
Newer Older
Federico Sismondi's avatar
Federico Sismondi committed
1 2
ioppytest framework:
--------------------
3

Federico Sismondi's avatar
Federico Sismondi committed
4
ioppytest is a framework for running interoperability tests.
5

Federico Sismondi's avatar
Federico Sismondi committed
6 7 8 9 10 11 12 13 14 15
This initial version tackles technical interoperability testing (CoAP
and 6LoWPAN interop tests), and effort is being made to implement
interop semantic tests notably for running tests in the WoT and OneM2M
context.

This repo contains all necessary software (and their dependencies) for
running a interoperability test sessions between two implementations
under test (IUT).

This can be run as standalone software and also integrated to f-interop
16 17 18
architecture.


Federico Sismondi's avatar
Federico Sismondi committed
19 20
Implemented test suites in the ioppytest framework:
---------------------------------------------------
21

Federico Sismondi's avatar
Federico Sismondi committed
22 23
The test suites implemented which are currently supported by the
framework are:
24

Federico Sismondi's avatar
Federico Sismondi committed
25 26 27
- CoAP Test Suite (user's IUT vs automated-IUT)
- CoAP Test Suite (between two users' IUT)
- 6LoWPAN Test Suite (between two users' IUT) (WIP)
28

29 30


Federico Sismondi's avatar
Federico Sismondi committed
31 32
Test setup:
-----------
33

Federico Sismondi's avatar
Federico Sismondi committed
34 35 36
An interop session happens between two implementation under test (IUT),
the following diagram shows the setup including the IUTs, testing tool
and auxiliary components needed for running a interop session.
37

Federico Sismondi's avatar
Federico Sismondi committed
38 39
All interactions between components take place using the AMQP event bus
(AMQP pub/sub mechanism)
40 41

```
Federico Sismondi's avatar
Federico Sismondi committed
42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86
                    +----------------------------+             +----------------------------+             +----------------------------+
                    |                            |             |                            |             |                            |
                    |    ioppytest Test Tool     |             |     User Interface         |             |     User Interface         |
                    |(CoAP, 6LoWPAN, OneM2M, etc)|             |         (user 1)           |             |         (user 2)           |
                    |                            |             |                            |             |                            |
                    |                            |             |                            |             |                            |
                    +----------------------------+             +----------------------------+             +----------------------------+

                             ^    +                                     ^    +                                     ^    +
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_x.tun.packet |    | toAgent.agent_y.tun.packet          |    |  ui.user1.step_verify.reply         |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_y.tun.packet |    | toAgent.agent_x.tun.packet          |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    | ui.user1.step_verify.request        |    |                                     |    |
                             +    v                                     +    v                                     +    v

        +------------------------------------------------------------------------------------------------------------------------------------------------>
                                                                         AMQP Event Bus
        <-------------------------------------------------------------------------------------------------------------------------------------------------+

                                                       +     ^                                        +     ^
                                                       |     | toAgent.agent_x.tun.packetet           |     |  fromAgent.agent_y.tun.packet
                              data.tun.toAgent.agent_x |     |                                        |     |
                                                       |     |              toAgent.agent_y.tun.packet|     |
                                                       v     |                                        v     |
                                PC        +------------+-----+-------------+              +-----------+-----+--------------+
                                user 1    |                                |              |                                |
                                          |      Agent (agent_x)           |              |      Agent (agent_y)           |
                                          |        (tun mode)              |              |        (tun mode)              |
                                          |                                |              |                                |
                                          |                                |              |                                |
                                          | +-----+tun interface+-------+  |              | +-----+tun interface+-------+  |
                                          |                                |              |                                |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          | |         IPv6+based         | |              | |         IPv6+based         | |
                                          | |        communicating       | |              | |        communicating       | |
                                          | |      piece of software     | |              | |      piece of software     | |
                                          | |      (e.g. coap client)    | |              | |      (e.g. coap server)    | |
                                          | |                            | |              | |                            | |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          |                                |              |                                |
                                          +--------------------------------+              +--------------------------------+
87 88
```

Federico Sismondi's avatar
Federico Sismondi committed
89 90 91 92 93 94 95 96 97
Event Bus API:
--------------

All the calls between the components are documented here:

[CORE API doc](http://doc.f-interop.eu/interop/)
and
[interop tests API doc](http://doc.f-interop.eu/interop/)

98 99


Federico Sismondi's avatar
Federico Sismondi committed
100 101
Running a test suite:
---------------------
102

Federico Sismondi's avatar
Federico Sismondi committed
103
user needs :
104

Federico Sismondi's avatar
Federico Sismondi committed
105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137
- an implementation under test (IUT) of a standard supported/protocol
by ioppytest framework, which either runs in some specific hardware or
locally in user's PC
- [the agent component](http://doc.f-interop.eu/interop/#agent)
which will route all the packets emitted from the IUT to the backend
and also to the second IUT (and viceversa)
- a user interface to help coordinating the tests
(either GUI or CLI component)

# Running a test suite using F-Interop platform

go to [go.f-interop.eu](go.f-interop.eu) and follow the instructions

# Running a test suite standalone

This mode of execution work for any of the following circustantces

- controls one IUT and wants to run tests against one of the
automated-IUTs the framework supports
- controls one IUT and is in direct contact with a second user
controlling a second IUT
- controls both implementations (IUTs) taking part in the interop
session

## Set up up the message broker

The interop testing tool use AMQP for sending messages between its
components, and the remote ones (like the agent). When running a
standalone setup the user first needs to have a RMQ broker running..

RMQ broker is a component which is **external** to the testing tool
and which establish the session and infrastructure so compomnents can
communicate with each other during the test session.
138

139 140 141 142 143 144 145
The options for this are:

- install locally RabbitMQ message broker on local machine,
create RMQ vhost, user, pass on local machine

    (# TODO add instructions)

Federico Sismondi's avatar
Federico Sismondi committed
146 147 148 149 150
- Request a remote vhost and credentials (user,pass) to
federico.sismondi@inria.fr (recommended)

don't hesitate to contact me, this is a very simple procedure and it's
free :D
151

Federico Sismondi's avatar
Federico Sismondi committed
152
## Export AMQP environment variables
153

154 155 156
after having a created vhost with its user/password,
export in the machine where the testing tool is running the following
env vars:
157

Federico Sismondi's avatar
Federico Sismondi committed
158 159 160 161
```
export AMQP_URL='amqp://someUser:somePassword@server/amqp_vhost'
export AMQP_EXCHANGE='amq.topic'
```
162

Federico Sismondi's avatar
Federico Sismondi committed
163 164 165 166 167
## Download the source code (see `Clonning the project` for more info)
```
git clone --recursive https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
cd ioppytest
```
168 169


Federico Sismondi's avatar
Federico Sismondi committed
170
## Build the testing tools
171

Federico Sismondi's avatar
Federico Sismondi committed
172 173 174 175
(docker, py2 and py3 needs to be installed in the machine)
```
make docker-build-all
```
176

Federico Sismondi's avatar
Federico Sismondi committed
177 178 179 180
## Run testing tool (CoAP testing tool example)
```
make run-coap-testing-tool
```
181

Federico Sismondi's avatar
Federico Sismondi committed
182 183 184 185
## Connect to the interop session using the CLI
```
make run-cli
```
186

Federico Sismondi's avatar
Federico Sismondi committed
187
## Connect the agent to the backend
188

Federico Sismondi's avatar
Federico Sismondi committed
189
if user's IUT is a CoAP client:
190

Federico Sismondi's avatar
Federico Sismondi committed
191 192 193
```
make run-agent-coap-client
```
194

Federico Sismondi's avatar
Federico Sismondi committed
195
if user's IUT is a CoAP server:
196

197
```
Federico Sismondi's avatar
Federico Sismondi committed
198
make run-agent-coap-server
199 200
```

Federico Sismondi's avatar
Federico Sismondi committed
201 202 203 204 205 206 207 208 209 210 211 212 213 214 215
## Running a second IUT

### User to user session, second user with his/her own IUT

The second IUT needs to connect to the same VHOST the same way first IUT
did. For this the RMQ broker needs to be reachable by this second IUT
 (and it's agent instance).

If this is the case then user 2 should just export the same environment
 variables as user 1, and launch agent, and CLI just as user 1 did.

### Single user session, agains an automated-IUT

 If the user wants to run test against one of the automated-IUT
 (~reference implementation) supported by ioppytest:
216

217
```
Federico Sismondi's avatar
Federico Sismondi committed
218
make run-coap-server
219 220
```

Federico Sismondi's avatar
Federico Sismondi committed
221 222
or for a coap client automated implementation:

223
```
Federico Sismondi's avatar
Federico Sismondi committed
224
make run-coap-client
225 226
```

Federico Sismondi's avatar
Federico Sismondi committed
227 228 229 230 231 232 233 234 235 236 237 238 239 240

## Running the interop session

Simply follow the CLI instructions and enjoy! :)


The implemented tests are based on this specification:
[ETSI CoAP test description](http://www.etsi.org/plugtests/CoAP/Document/CoAP_TestDescriptions_v015.pdf)


Developping a new test suite:
-----------------------------

## Clonning the project
241
```
Federico Sismondi's avatar
Federico Sismondi committed
242 243
git clone --recursive https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
cd ioppytest
244 245
```

Federico Sismondi's avatar
Federico Sismondi committed
246
### Attention with the git submodules!
247

Federico Sismondi's avatar
Federico Sismondi committed
248
remember when cloning a project with submodules to use --recursive flag
249
```
Federico Sismondi's avatar
Federico Sismondi committed
250
git clone --recursive ...
251 252
```

Federico Sismondi's avatar
Federico Sismondi committed
253
or else (in case you forgot about the flag), right after cloning you can:
254
```
Federico Sismondi's avatar
Federico Sismondi committed
255
git submodule update --init --recursive
256 257
```

Federico Sismondi's avatar
Federico Sismondi committed
258 259 260 261 262 263
whenever you find that your utils libraries are not the latests versions
you can 'bring' those last changes from the main utils repo to your project
with:
```
git submodule update --remote --merge
```
264

Federico Sismondi's avatar
Federico Sismondi committed
265
after bringing the last changes you can update your project with the last changes by doing:
266
```
Federico Sismondi's avatar
Federico Sismondi committed
267 268 269
git add <someSubModuleDir>
git commit -m 'updated submodule reference to last commit'
git push
270 271
```

Federico Sismondi's avatar
Federico Sismondi committed
272 273 274 275 276 277 278 279 280 281 282

## How to merge new features to upstream branch ?

Read CONTRIBUTING.rst document

## How can I develop & debug test suites?

for getting logs from the docker containers you can:
```
make get-logs
```
283

284
for running coap_testing_tool manually from docker api:
Federico Sismondi's avatar
Federico Sismondi committed
285

286
```
Federico Sismondi's avatar
Federico Sismondi committed
287
docker run -it
Federico Sismondi's avatar
Federico Sismondi committed
288 289 290 291 292
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
293 294
```

295 296 297

alternatively, if you are curious and you want to know
what's under the hood:
Federico Sismondi's avatar
Federico Sismondi committed
298

299
```
Federico Sismondi's avatar
Federico Sismondi committed
300 301 302 303 304 305 306 307
docker run -it
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
    bash

308 309 310 311 312 313 314 315 316 317 318 319
root@bab3b2220510:/coap_testing_tool# supervisord -c supervisor.conf
root@bab3b2220510:/coap_testing_tool# supervisorctl -c supervisor.conf
agent                            RUNNING   pid 28, uptime 0:00:02
automated-iut                    STARTING
bootstrap-agent-TT               RUNNING   pid 19, uptime 0:00:02
packet-router                    RUNNING   pid 24, uptime 0:00:02
packet-sniffer                   RUNNING   pid 18, uptime 0:00:02
tat                              RUNNING   pid 17, uptime 0:00:02
test-coordinator                 RUNNING   pid 26, uptime 0:00:02
supervisor>
```

Federico Sismondi's avatar
Federico Sismondi committed
320 321
or you can also run directly the processes without docker:
(supervisord needed)
322

Federico Sismondi's avatar
Federico Sismondi committed
323 324 325 326
```
sudo -E supervisord -c routeToConfigurationFileForTheTestSuite
sudo supervisorctl -c routeToConfigurationFileForTheTestSuite
```
327

Federico Sismondi's avatar
Federico Sismondi committed
328 329
note: use -E when launching supervisor process, it preserves the
env vars
330 331


Federico Sismondi's avatar
Federico Sismondi committed
332 333
FAQ
---
334

Federico Sismondi's avatar
Federico Sismondi committed
335
- How can I install docker on my machine?
336

Federico Sismondi's avatar
Federico Sismondi committed
337
    For this just follow this instructions: https://docs.docker.com/engine/installation/
338

339

Federico Sismondi's avatar
Federico Sismondi committed
340 341 342 343
- How do I install supervisord on my machine?
    Install supervisor (needed for spawning and monitoring processes):
    For this follow this instructions:
    http://supervisord.org/installing.html
344

Federico Sismondi's avatar
Federico Sismondi committed
345 346
- I have my own CoAP implementation, how can I add it as an
automated-IUT into CoAP Testing Tool:
347

Federico Sismondi's avatar
Federico Sismondi committed
348
    please contact federico.sismondi@inria.fr
349 350


Federico Sismondi's avatar
Federico Sismondi committed
351
- Docker build returns a "cannot fetch package" or a "cannot resolve .."
352

Federico Sismondi's avatar
Federico Sismondi committed
353
    try using ```--no-cache``` for the docker build
354

Federico Sismondi's avatar
Federico Sismondi committed
355
    more info http://stackoverflow.com/questions/24991136/docker-build-could-not-resolve-archive-ubuntu-com-apt-get-fails-to-install-a