README.md 13.5 KB
Newer Older
Federico Sismondi's avatar
Federico Sismondi committed
1 2
ioppytest framework:
--------------------
3

Federico Sismondi's avatar
Federico Sismondi committed
4
ioppytest is a framework for running interoperability tests.
5

Federico Sismondi's avatar
Federico Sismondi committed
6 7 8 9 10 11 12 13 14 15
This initial version tackles technical interoperability testing (CoAP
and 6LoWPAN interop tests), and effort is being made to implement
interop semantic tests notably for running tests in the WoT and OneM2M
context.

This repo contains all necessary software (and their dependencies) for
running a interoperability test sessions between two implementations
under test (IUT).

This can be run as standalone software and also integrated to f-interop
16 17 18
architecture.


Federico Sismondi's avatar
Federico Sismondi committed
19 20
Implemented test suites in the ioppytest framework:
---------------------------------------------------
21

Federico Sismondi's avatar
Federico Sismondi committed
22 23
The test suites implemented which are currently supported by the
framework are:
24

Federico Sismondi's avatar
Federico Sismondi committed
25 26 27
- CoAP Test Suite (user's IUT vs automated-IUT)
- CoAP Test Suite (between two users' IUT)
- 6LoWPAN Test Suite (between two users' IUT) (WIP)
28

29 30


Federico Sismondi's avatar
Federico Sismondi committed
31 32
Test setup:
-----------
33

Federico Sismondi's avatar
Federico Sismondi committed
34 35 36
An interop session happens between two implementation under test (IUT),
the following diagram shows the setup including the IUTs, testing tool
and auxiliary components needed for running a interop session.
37

Federico Sismondi's avatar
Federico Sismondi committed
38 39
All interactions between components take place using the AMQP event bus
(AMQP pub/sub mechanism)
40 41

```
Federico Sismondi's avatar
Federico Sismondi committed
42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86
                    +----------------------------+             +----------------------------+             +----------------------------+
                    |                            |             |                            |             |                            |
                    |    ioppytest Test Tool     |             |     User Interface         |             |     User Interface         |
                    |(CoAP, 6LoWPAN, OneM2M, etc)|             |         (user 1)           |             |         (user 2)           |
                    |                            |             |                            |             |                            |
                    |                            |             |                            |             |                            |
                    +----------------------------+             +----------------------------+             +----------------------------+

                             ^    +                                     ^    +                                     ^    +
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_x.tun.packet |    | toAgent.agent_y.tun.packet          |    |  ui.user1.step_verify.reply         |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_y.tun.packet |    | toAgent.agent_x.tun.packet          |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    | ui.user1.step_verify.request        |    |                                     |    |
                             +    v                                     +    v                                     +    v

        +------------------------------------------------------------------------------------------------------------------------------------------------>
                                                                         AMQP Event Bus
        <-------------------------------------------------------------------------------------------------------------------------------------------------+

                                                       +     ^                                        +     ^
                                                       |     | toAgent.agent_x.tun.packetet           |     |  fromAgent.agent_y.tun.packet
                              data.tun.toAgent.agent_x |     |                                        |     |
                                                       |     |              toAgent.agent_y.tun.packet|     |
                                                       v     |                                        v     |
                                PC        +------------+-----+-------------+              +-----------+-----+--------------+
                                user 1    |                                |              |                                |
                                          |      Agent (agent_x)           |              |      Agent (agent_y)           |
                                          |        (tun mode)              |              |        (tun mode)              |
                                          |                                |              |                                |
                                          |                                |              |                                |
                                          | +-----+tun interface+-------+  |              | +-----+tun interface+-------+  |
                                          |                                |              |                                |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          | |         IPv6+based         | |              | |         IPv6+based         | |
                                          | |        communicating       | |              | |        communicating       | |
                                          | |      piece of software     | |              | |      piece of software     | |
                                          | |      (e.g. coap client)    | |              | |      (e.g. coap server)    | |
                                          | |                            | |              | |                            | |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          |                                |              |                                |
                                          +--------------------------------+              +--------------------------------+
87 88
```

Federico Sismondi's avatar
Federico Sismondi committed
89 90 91 92 93 94 95 96 97
Event Bus API:
--------------

All the calls between the components are documented here:

[CORE API doc](http://doc.f-interop.eu/interop/)
and
[interop tests API doc](http://doc.f-interop.eu/interop/)

98 99


Federico Sismondi's avatar
Federico Sismondi committed
100 101
Running a test suite:
---------------------
102

Federico Sismondi's avatar
Federico Sismondi committed
103
user needs :
104

Federico Sismondi's avatar
Federico Sismondi committed
105 106
- an implementation under test (IUT) of a standard supported/protocol
by ioppytest framework, which either runs in some specific hardware or
Federico Sismondi's avatar
Federico Sismondi committed
107
locally in user's PC, e.g. a coap client implementation
108
-  run
Federico Sismondi's avatar
Federico Sismondi committed
109
[the agent component](http://doc.f-interop.eu/interop/#agent)
Federico Sismondi's avatar
Federico Sismondi committed
110 111 112 113 114
which will route all the packets emitted from the IUT to the backend
and also to the second IUT (and viceversa)
- a user interface to help coordinating the tests
(either GUI or CLI component)

115
For simplifying the access to CLI, agent and other components, ioppytest
116 117 118
inlcudes a Makefile, with it you can use `make cmd`,
for more information execute `make help`

Federico Sismondi's avatar
Federico Sismondi committed
119 120 121 122 123 124
# Running a test suite using F-Interop platform

go to [go.f-interop.eu](go.f-interop.eu) and follow the instructions

# Running a test suite standalone

Federico Sismondi's avatar
Federico Sismondi committed
125
This mode of execution work for any of the following circumstances
Federico Sismondi's avatar
Federico Sismondi committed
126

Federico Sismondi's avatar
Federico Sismondi committed
127
- user controls one IUT and wants to run tests against one of the
Federico Sismondi's avatar
Federico Sismondi committed
128
automated-IUTs the framework supports
Federico Sismondi's avatar
Federico Sismondi committed
129
- user controls one IUT and is in direct contact with a second user
Federico Sismondi's avatar
Federico Sismondi committed
130
controlling a second IUT
Federico Sismondi's avatar
Federico Sismondi committed
131
- user controls both implementations (IUTs) taking part in the interop
Federico Sismondi's avatar
Federico Sismondi committed
132 133 134 135 136 137 138 139 140 141 142
session

## Set up up the message broker

The interop testing tool use AMQP for sending messages between its
components, and the remote ones (like the agent). When running a
standalone setup the user first needs to have a RMQ broker running..

RMQ broker is a component which is **external** to the testing tool
and which establish the session and infrastructure so compomnents can
communicate with each other during the test session.
143

144 145 146 147 148 149 150
The options for this are:

- install locally RabbitMQ message broker on local machine,
create RMQ vhost, user, pass on local machine

    (# TODO add instructions)

Federico Sismondi's avatar
Federico Sismondi committed
151 152 153 154 155
- Request a remote vhost and credentials (user,pass) to
federico.sismondi@inria.fr (recommended)

don't hesitate to contact me, this is a very simple procedure and it's
free :D
156

Federico Sismondi's avatar
Federico Sismondi committed
157
## Export AMQP environment variables
158

159 160 161
after having a created vhost with its user/password,
export in the machine where the testing tool is running the following
env vars:
162

Federico Sismondi's avatar
Federico Sismondi committed
163 164 165 166
```
export AMQP_URL='amqp://someUser:somePassword@server/amqp_vhost'
export AMQP_EXCHANGE='amq.topic'
```
167

Federico Sismondi's avatar
Federico Sismondi committed
168 169 170 171 172
## Download the source code (see `Clonning the project` for more info)
```
git clone --recursive https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
cd ioppytest
```
173 174


Federico Sismondi's avatar
Federico Sismondi committed
175
## Build the testing tools
176

Federico Sismondi's avatar
Federico Sismondi committed
177 178 179 180
(docker, py2 and py3 needs to be installed in the machine)
```
make docker-build-all
```
181

Federico Sismondi's avatar
Federico Sismondi committed
182 183 184 185
## Run testing tool (CoAP testing tool example)
```
make run-coap-testing-tool
```
186

Federico Sismondi's avatar
Federico Sismondi committed
187 188 189 190
## Connect to the interop session using the CLI
```
make run-cli
```
191

Federico Sismondi's avatar
Federico Sismondi committed
192
## Connect the agent to the backend
193

Federico Sismondi's avatar
Federico Sismondi committed
194
if user's IUT is a CoAP client:
195

Federico Sismondi's avatar
Federico Sismondi committed
196 197 198
```
make run-agent-coap-client
```
199

Federico Sismondi's avatar
Federico Sismondi committed
200
if user's IUT is a CoAP server:
201

202
```
Federico Sismondi's avatar
Federico Sismondi committed
203
make run-agent-coap-server
204 205
```

Federico Sismondi's avatar
Federico Sismondi committed
206 207 208 209 210 211 212 213 214 215 216
## Running a second IUT

### User to user session, second user with his/her own IUT

The second IUT needs to connect to the same VHOST the same way first IUT
did. For this the RMQ broker needs to be reachable by this second IUT
 (and it's agent instance).

If this is the case then user 2 should just export the same environment
 variables as user 1, and launch agent, and CLI just as user 1 did.

217
### Single user session, against an automated-IUT
Federico Sismondi's avatar
Federico Sismondi committed
218 219 220

 If the user wants to run test against one of the automated-IUT
 (~reference implementation) supported by ioppytest:
221

222
```
Federico Sismondi's avatar
Federico Sismondi committed
223
make run-coap-server
224 225
```

Federico Sismondi's avatar
Federico Sismondi committed
226 227
or for a coap client automated implementation:

228
```
Federico Sismondi's avatar
Federico Sismondi committed
229
make run-coap-client
230 231
```

Federico Sismondi's avatar
Federico Sismondi committed
232 233 234 235 236 237 238 239 240 241 242 243 244 245

## Running the interop session

Simply follow the CLI instructions and enjoy! :)


The implemented tests are based on this specification:
[ETSI CoAP test description](http://www.etsi.org/plugtests/CoAP/Document/CoAP_TestDescriptions_v015.pdf)


Developping a new test suite:
-----------------------------

## Clonning the project
246
```
Federico Sismondi's avatar
Federico Sismondi committed
247 248
git clone --recursive https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
cd ioppytest
249 250
```

Federico Sismondi's avatar
Federico Sismondi committed
251
### Attention with the git submodules!
252

Federico Sismondi's avatar
Federico Sismondi committed
253
remember when cloning a project with submodules to use --recursive flag
254
```
Federico Sismondi's avatar
Federico Sismondi committed
255
git clone --recursive ...
256 257
```

Federico Sismondi's avatar
Federico Sismondi committed
258
or else (in case you forgot about the flag), right after cloning you can:
259
```
Federico Sismondi's avatar
Federico Sismondi committed
260
git submodule update --init --recursive
261 262
```

Federico Sismondi's avatar
Federico Sismondi committed
263 264 265 266 267 268
whenever you find that your utils libraries are not the latests versions
you can 'bring' those last changes from the main utils repo to your project
with:
```
git submodule update --remote --merge
```
269

Federico Sismondi's avatar
Federico Sismondi committed
270
after bringing the last changes you can update your project with the last changes by doing:
271
```
Federico Sismondi's avatar
Federico Sismondi committed
272 273 274
git add <someSubModuleDir>
git commit -m 'updated submodule reference to last commit'
git push
275 276
```

Federico Sismondi's avatar
Federico Sismondi committed
277 278 279 280 281 282 283 284 285 286 287

## How to merge new features to upstream branch ?

Read CONTRIBUTING.rst document

## How can I develop & debug test suites?

for getting logs from the docker containers you can:
```
make get-logs
```
288

289
for running coap_testing_tool manually from docker api:
Federico Sismondi's avatar
Federico Sismondi committed
290

291
```
Federico Sismondi's avatar
Federico Sismondi committed
292
docker run -it
Federico Sismondi's avatar
Federico Sismondi committed
293 294 295 296 297
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
298 299
```

300 301 302

alternatively, if you are curious and you want to know
what's under the hood:
Federico Sismondi's avatar
Federico Sismondi committed
303

304
```
Federico Sismondi's avatar
Federico Sismondi committed
305 306 307 308 309 310 311 312
docker run -it
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
    bash

313 314
root@bab3b2220510:/ioppytest# supervisord -c supervisor.conf
root@bab3b2220510:/ioppytest# supervisorctl -c supervisor.conf
315 316 317 318 319 320 321 322 323
agent                            RUNNING   pid 28, uptime 0:00:02
bootstrap-agent-TT               RUNNING   pid 19, uptime 0:00:02
packet-router                    RUNNING   pid 24, uptime 0:00:02
packet-sniffer                   RUNNING   pid 18, uptime 0:00:02
tat                              RUNNING   pid 17, uptime 0:00:02
test-coordinator                 RUNNING   pid 26, uptime 0:00:02
supervisor>
```

Federico Sismondi's avatar
Federico Sismondi committed
324 325
or you can also run directly the processes without docker:
(supervisord needed)
326

Federico Sismondi's avatar
Federico Sismondi committed
327 328
```
sudo -E supervisord -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
329
sudo -E supervisorctl -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
330
```
331

Federico Sismondi's avatar
Federico Sismondi committed
332 333 334
you can use for example envs/coap_testing_tool/supervisor.conf.ini
for using the coap_testing_tool

Federico Sismondi's avatar
Federico Sismondi committed
335 336
note: use -E when launching supervisor process, it preserves the
env vars
337 338


Federico Sismondi's avatar
Federico Sismondi committed
339 340
FAQ
---
341

Federico Sismondi's avatar
Federico Sismondi committed
342
- How can I install docker on my machine?
343

Federico Sismondi's avatar
Federico Sismondi committed
344
    For this just follow this instructions: https://docs.docker.com/engine/installation/
345

346

Federico Sismondi's avatar
Federico Sismondi committed
347 348 349 350
- How do I install supervisord on my machine?
    Install supervisor (needed for spawning and monitoring processes):
    For this follow this instructions:
    http://supervisord.org/installing.html
351

Federico Sismondi's avatar
Federico Sismondi committed
352 353
- I have my own CoAP implementation, how can I add it as an
automated-IUT into CoAP Testing Tool:
354

Federico Sismondi's avatar
Federico Sismondi committed
355
    please contact federico.sismondi@inria.fr
356 357


Federico Sismondi's avatar
Federico Sismondi committed
358
- Docker build returns a "cannot fetch package" or a "cannot resolve .."
359

Federico Sismondi's avatar
Federico Sismondi committed
360
    try using ```--no-cache``` for the docker build
361

Federico Sismondi's avatar
Federico Sismondi committed
362
    more info http://stackoverflow.com/questions/24991136/docker-build-could-not-resolve-archive-ubuntu-com-apt-get-fails-to-install-a