README.md 13.4 KB
Newer Older
Federico Sismondi's avatar
Federico Sismondi committed
1 2
ioppytest framework:
--------------------
3

Federico Sismondi's avatar
Federico Sismondi committed
4
ioppytest is a framework for running interoperability tests.
5

Federico Sismondi's avatar
Federico Sismondi committed
6 7 8 9 10 11 12 13 14 15
This initial version tackles technical interoperability testing (CoAP
and 6LoWPAN interop tests), and effort is being made to implement
interop semantic tests notably for running tests in the WoT and OneM2M
context.

This repo contains all necessary software (and their dependencies) for
running a interoperability test sessions between two implementations
under test (IUT).

This can be run as standalone software and also integrated to f-interop
16 17 18
architecture.


Federico Sismondi's avatar
Federico Sismondi committed
19 20
Implemented test suites in the ioppytest framework:
---------------------------------------------------
21

Federico Sismondi's avatar
Federico Sismondi committed
22 23
The test suites implemented which are currently supported by the
framework are:
24

Federico Sismondi's avatar
Federico Sismondi committed
25 26 27
- CoAP Test Suite (user's IUT vs automated-IUT)
- CoAP Test Suite (between two users' IUT)
- 6LoWPAN Test Suite (between two users' IUT) (WIP)
28

29 30


Federico Sismondi's avatar
Federico Sismondi committed
31 32
Test setup:
-----------
33

Federico Sismondi's avatar
Federico Sismondi committed
34 35 36
An interop session happens between two implementation under test (IUT),
the following diagram shows the setup including the IUTs, testing tool
and auxiliary components needed for running a interop session.
37

Federico Sismondi's avatar
Federico Sismondi committed
38 39
All interactions between components take place using the AMQP event bus
(AMQP pub/sub mechanism)
40 41

```
Federico Sismondi's avatar
Federico Sismondi committed
42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86
                    +----------------------------+             +----------------------------+             +----------------------------+
                    |                            |             |                            |             |                            |
                    |    ioppytest Test Tool     |             |     User Interface         |             |     User Interface         |
                    |(CoAP, 6LoWPAN, OneM2M, etc)|             |         (user 1)           |             |         (user 2)           |
                    |                            |             |                            |             |                            |
                    |                            |             |                            |             |                            |
                    +----------------------------+             +----------------------------+             +----------------------------+

                             ^    +                                     ^    +                                     ^    +
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_x.tun.packet |    | toAgent.agent_y.tun.packet          |    |  ui.user1.step_verify.reply         |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_y.tun.packet |    | toAgent.agent_x.tun.packet          |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    | ui.user1.step_verify.request        |    |                                     |    |
                             +    v                                     +    v                                     +    v

        +------------------------------------------------------------------------------------------------------------------------------------------------>
                                                                         AMQP Event Bus
        <-------------------------------------------------------------------------------------------------------------------------------------------------+

                                                       +     ^                                        +     ^
                                                       |     | toAgent.agent_x.tun.packetet           |     |  fromAgent.agent_y.tun.packet
                              data.tun.toAgent.agent_x |     |                                        |     |
                                                       |     |              toAgent.agent_y.tun.packet|     |
                                                       v     |                                        v     |
                                PC        +------------+-----+-------------+              +-----------+-----+--------------+
                                user 1    |                                |              |                                |
                                          |      Agent (agent_x)           |              |      Agent (agent_y)           |
                                          |        (tun mode)              |              |        (tun mode)              |
                                          |                                |              |                                |
                                          |                                |              |                                |
                                          | +-----+tun interface+-------+  |              | +-----+tun interface+-------+  |
                                          |                                |              |                                |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          | |         IPv6+based         | |              | |         IPv6+based         | |
                                          | |        communicating       | |              | |        communicating       | |
                                          | |      piece of software     | |              | |      piece of software     | |
                                          | |      (e.g. coap client)    | |              | |      (e.g. coap server)    | |
                                          | |                            | |              | |                            | |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          |                                |              |                                |
                                          +--------------------------------+              +--------------------------------+
87 88
```

Federico Sismondi's avatar
Federico Sismondi committed
89 90 91 92 93 94 95 96 97
Event Bus API:
--------------

All the calls between the components are documented here:

[CORE API doc](http://doc.f-interop.eu/interop/)
and
[interop tests API doc](http://doc.f-interop.eu/interop/)

98 99


Federico Sismondi's avatar
Federico Sismondi committed
100 101
Running a test suite:
---------------------
102

Federico Sismondi's avatar
Federico Sismondi committed
103
user needs :
104

Federico Sismondi's avatar
Federico Sismondi committed
105 106
- an implementation under test (IUT) of a standard supported/protocol
by ioppytest framework, which either runs in some specific hardware or
Federico Sismondi's avatar
Federico Sismondi committed
107 108 109
locally in user's PC, e.g. a coap client implementation
- download and run
[the agent component](http://doc.f-interop.eu/interop/#agent)
Federico Sismondi's avatar
Federico Sismondi committed
110 111 112 113 114 115 116 117 118 119 120
which will route all the packets emitted from the IUT to the backend
and also to the second IUT (and viceversa)
- a user interface to help coordinating the tests
(either GUI or CLI component)

# Running a test suite using F-Interop platform

go to [go.f-interop.eu](go.f-interop.eu) and follow the instructions

# Running a test suite standalone

Federico Sismondi's avatar
Federico Sismondi committed
121
This mode of execution work for any of the following circumstances
Federico Sismondi's avatar
Federico Sismondi committed
122

Federico Sismondi's avatar
Federico Sismondi committed
123
- user controls one IUT and wants to run tests against one of the
Federico Sismondi's avatar
Federico Sismondi committed
124
automated-IUTs the framework supports
Federico Sismondi's avatar
Federico Sismondi committed
125
- user controls one IUT and is in direct contact with a second user
Federico Sismondi's avatar
Federico Sismondi committed
126
controlling a second IUT
Federico Sismondi's avatar
Federico Sismondi committed
127
- user controls both implementations (IUTs) taking part in the interop
Federico Sismondi's avatar
Federico Sismondi committed
128 129 130 131 132 133 134 135 136 137 138
session

## Set up up the message broker

The interop testing tool use AMQP for sending messages between its
components, and the remote ones (like the agent). When running a
standalone setup the user first needs to have a RMQ broker running..

RMQ broker is a component which is **external** to the testing tool
and which establish the session and infrastructure so compomnents can
communicate with each other during the test session.
139

140 141 142 143 144 145 146
The options for this are:

- install locally RabbitMQ message broker on local machine,
create RMQ vhost, user, pass on local machine

    (# TODO add instructions)

Federico Sismondi's avatar
Federico Sismondi committed
147 148 149 150 151
- Request a remote vhost and credentials (user,pass) to
federico.sismondi@inria.fr (recommended)

don't hesitate to contact me, this is a very simple procedure and it's
free :D
152

Federico Sismondi's avatar
Federico Sismondi committed
153
## Export AMQP environment variables
154

155 156 157
after having a created vhost with its user/password,
export in the machine where the testing tool is running the following
env vars:
158

Federico Sismondi's avatar
Federico Sismondi committed
159 160 161 162
```
export AMQP_URL='amqp://someUser:somePassword@server/amqp_vhost'
export AMQP_EXCHANGE='amq.topic'
```
163

Federico Sismondi's avatar
Federico Sismondi committed
164 165 166 167 168
## Download the source code (see `Clonning the project` for more info)
```
git clone --recursive https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
cd ioppytest
```
169 170


Federico Sismondi's avatar
Federico Sismondi committed
171
## Build the testing tools
172

Federico Sismondi's avatar
Federico Sismondi committed
173 174 175 176
(docker, py2 and py3 needs to be installed in the machine)
```
make docker-build-all
```
177

Federico Sismondi's avatar
Federico Sismondi committed
178 179 180 181
## Run testing tool (CoAP testing tool example)
```
make run-coap-testing-tool
```
182

Federico Sismondi's avatar
Federico Sismondi committed
183 184 185 186
## Connect to the interop session using the CLI
```
make run-cli
```
187

Federico Sismondi's avatar
Federico Sismondi committed
188
## Connect the agent to the backend
189

Federico Sismondi's avatar
Federico Sismondi committed
190
if user's IUT is a CoAP client:
191

Federico Sismondi's avatar
Federico Sismondi committed
192 193 194
```
make run-agent-coap-client
```
195

Federico Sismondi's avatar
Federico Sismondi committed
196
if user's IUT is a CoAP server:
197

198
```
Federico Sismondi's avatar
Federico Sismondi committed
199
make run-agent-coap-server
200 201
```

Federico Sismondi's avatar
Federico Sismondi committed
202 203 204 205 206 207 208 209 210 211 212 213 214 215 216
## Running a second IUT

### User to user session, second user with his/her own IUT

The second IUT needs to connect to the same VHOST the same way first IUT
did. For this the RMQ broker needs to be reachable by this second IUT
 (and it's agent instance).

If this is the case then user 2 should just export the same environment
 variables as user 1, and launch agent, and CLI just as user 1 did.

### Single user session, agains an automated-IUT

 If the user wants to run test against one of the automated-IUT
 (~reference implementation) supported by ioppytest:
217

218
```
Federico Sismondi's avatar
Federico Sismondi committed
219
make run-coap-server
220 221
```

Federico Sismondi's avatar
Federico Sismondi committed
222 223
or for a coap client automated implementation:

224
```
Federico Sismondi's avatar
Federico Sismondi committed
225
make run-coap-client
226 227
```

Federico Sismondi's avatar
Federico Sismondi committed
228 229 230 231 232 233 234 235 236 237 238 239 240 241

## Running the interop session

Simply follow the CLI instructions and enjoy! :)


The implemented tests are based on this specification:
[ETSI CoAP test description](http://www.etsi.org/plugtests/CoAP/Document/CoAP_TestDescriptions_v015.pdf)


Developping a new test suite:
-----------------------------

## Clonning the project
242
```
Federico Sismondi's avatar
Federico Sismondi committed
243 244
git clone --recursive https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
cd ioppytest
245 246
```

Federico Sismondi's avatar
Federico Sismondi committed
247
### Attention with the git submodules!
248

Federico Sismondi's avatar
Federico Sismondi committed
249
remember when cloning a project with submodules to use --recursive flag
250
```
Federico Sismondi's avatar
Federico Sismondi committed
251
git clone --recursive ...
252 253
```

Federico Sismondi's avatar
Federico Sismondi committed
254
or else (in case you forgot about the flag), right after cloning you can:
255
```
Federico Sismondi's avatar
Federico Sismondi committed
256
git submodule update --init --recursive
257 258
```

Federico Sismondi's avatar
Federico Sismondi committed
259 260 261 262 263 264
whenever you find that your utils libraries are not the latests versions
you can 'bring' those last changes from the main utils repo to your project
with:
```
git submodule update --remote --merge
```
265

Federico Sismondi's avatar
Federico Sismondi committed
266
after bringing the last changes you can update your project with the last changes by doing:
267
```
Federico Sismondi's avatar
Federico Sismondi committed
268 269 270
git add <someSubModuleDir>
git commit -m 'updated submodule reference to last commit'
git push
271 272
```

Federico Sismondi's avatar
Federico Sismondi committed
273 274 275 276 277 278 279 280 281 282 283

## How to merge new features to upstream branch ?

Read CONTRIBUTING.rst document

## How can I develop & debug test suites?

for getting logs from the docker containers you can:
```
make get-logs
```
284

285
for running coap_testing_tool manually from docker api:
Federico Sismondi's avatar
Federico Sismondi committed
286

287
```
Federico Sismondi's avatar
Federico Sismondi committed
288
docker run -it
Federico Sismondi's avatar
Federico Sismondi committed
289 290 291 292 293
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
294 295
```

296 297 298

alternatively, if you are curious and you want to know
what's under the hood:
Federico Sismondi's avatar
Federico Sismondi committed
299

300
```
Federico Sismondi's avatar
Federico Sismondi committed
301 302 303 304 305 306 307 308
docker run -it
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
    bash

309 310
root@bab3b2220510:/ioppytest# supervisord -c supervisor.conf
root@bab3b2220510:/ioppytest# supervisorctl -c supervisor.conf
311 312 313 314 315 316 317 318 319 320
agent                            RUNNING   pid 28, uptime 0:00:02
automated-iut                    STARTING
bootstrap-agent-TT               RUNNING   pid 19, uptime 0:00:02
packet-router                    RUNNING   pid 24, uptime 0:00:02
packet-sniffer                   RUNNING   pid 18, uptime 0:00:02
tat                              RUNNING   pid 17, uptime 0:00:02
test-coordinator                 RUNNING   pid 26, uptime 0:00:02
supervisor>
```

Federico Sismondi's avatar
Federico Sismondi committed
321 322
or you can also run directly the processes without docker:
(supervisord needed)
323

Federico Sismondi's avatar
Federico Sismondi committed
324 325
```
sudo -E supervisord -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
326
sudo -E supervisorctl -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
327
```
328

Federico Sismondi's avatar
Federico Sismondi committed
329 330 331
you can use for example envs/coap_testing_tool/supervisor.conf.ini
for using the coap_testing_tool

Federico Sismondi's avatar
Federico Sismondi committed
332 333
note: use -E when launching supervisor process, it preserves the
env vars
334 335


Federico Sismondi's avatar
Federico Sismondi committed
336 337
FAQ
---
338

Federico Sismondi's avatar
Federico Sismondi committed
339
- How can I install docker on my machine?
340

Federico Sismondi's avatar
Federico Sismondi committed
341
    For this just follow this instructions: https://docs.docker.com/engine/installation/
342

343

Federico Sismondi's avatar
Federico Sismondi committed
344 345 346 347
- How do I install supervisord on my machine?
    Install supervisor (needed for spawning and monitoring processes):
    For this follow this instructions:
    http://supervisord.org/installing.html
348

Federico Sismondi's avatar
Federico Sismondi committed
349 350
- I have my own CoAP implementation, how can I add it as an
automated-IUT into CoAP Testing Tool:
351

Federico Sismondi's avatar
Federico Sismondi committed
352
    please contact federico.sismondi@inria.fr
353 354


Federico Sismondi's avatar
Federico Sismondi committed
355
- Docker build returns a "cannot fetch package" or a "cannot resolve .."
356

Federico Sismondi's avatar
Federico Sismondi committed
357
    try using ```--no-cache``` for the docker build
358

Federico Sismondi's avatar
Federico Sismondi committed
359
    more info http://stackoverflow.com/questions/24991136/docker-build-could-not-resolve-archive-ubuntu-com-apt-get-fails-to-install-a