README.md 13.2 KB
Newer Older
Federico Sismondi's avatar
Federico Sismondi committed
1
2
ioppytest framework:
--------------------
Federico Sismondi's avatar
Federico Sismondi committed
3

Federico Sismondi's avatar
Federico Sismondi committed
4
ioppytest is a framework for running interoperability tests.
Federico Sismondi's avatar
Federico Sismondi committed
5

Federico Sismondi's avatar
Federico Sismondi committed
6
7
8
9
10
11
12
13
14
15
This initial version tackles technical interoperability testing (CoAP
and 6LoWPAN interop tests), and effort is being made to implement
interop semantic tests notably for running tests in the WoT and OneM2M
context.

This repo contains all necessary software (and their dependencies) for
running a interoperability test sessions between two implementations
under test (IUT).

This can be run as standalone software and also integrated to f-interop
Federico Sismondi's avatar
Federico Sismondi committed
16
17
18
architecture.


Federico Sismondi's avatar
Federico Sismondi committed
19
20
Implemented test suites in the ioppytest framework:
---------------------------------------------------
Federico Sismondi's avatar
Federico Sismondi committed
21

Federico Sismondi's avatar
Federico Sismondi committed
22
23
The test suites implemented which are currently supported by the
framework are:
Federico Sismondi's avatar
Federico Sismondi committed
24

Federico Sismondi's avatar
Federico Sismondi committed
25
26
27
- CoAP Test Suite (user's IUT vs automated-IUT)
- CoAP Test Suite (between two users' IUT)
- 6LoWPAN Test Suite (between two users' IUT) (WIP)
Federico Sismondi's avatar
Federico Sismondi committed
28

29
30


Federico Sismondi's avatar
Federico Sismondi committed
31
32
Test setup:
-----------
Federico Sismondi's avatar
Federico Sismondi committed
33

Federico Sismondi's avatar
Federico Sismondi committed
34
35
36
An interop session happens between two implementation under test (IUT),
the following diagram shows the setup including the IUTs, testing tool
and auxiliary components needed for running a interop session.
37

Federico Sismondi's avatar
Federico Sismondi committed
38
39
All interactions between components take place using the AMQP event bus
(AMQP pub/sub mechanism)
40
41

```
Federico Sismondi's avatar
Federico Sismondi committed
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
                    +----------------------------+             +----------------------------+             +----------------------------+
                    |                            |             |                            |             |                            |
                    |    ioppytest Test Tool     |             |     User Interface         |             |     User Interface         |
                    |(CoAP, 6LoWPAN, OneM2M, etc)|             |         (user 1)           |             |         (user 2)           |
                    |                            |             |                            |             |                            |
                    |                            |             |                            |             |                            |
                    +----------------------------+             +----------------------------+             +----------------------------+

                             ^    +                                     ^    +                                     ^    +
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_x.tun.packet |    | toAgent.agent_y.tun.packet          |    |  ui.user1.step_verify.reply         |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_y.tun.packet |    | toAgent.agent_x.tun.packet          |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    | ui.user1.step_verify.request        |    |                                     |    |
                             +    v                                     +    v                                     +    v

        +------------------------------------------------------------------------------------------------------------------------------------------------>
                                                                         AMQP Event Bus
        <-------------------------------------------------------------------------------------------------------------------------------------------------+

                                                       +     ^                                        +     ^
                                                       |     | toAgent.agent_x.tun.packetet           |     |  fromAgent.agent_y.tun.packet
                              data.tun.toAgent.agent_x |     |                                        |     |
                                                       |     |              toAgent.agent_y.tun.packet|     |
                                                       v     |                                        v     |
                                PC        +------------+-----+-------------+              +-----------+-----+--------------+
                                user 1    |                                |              |                                |
                                          |      Agent (agent_x)           |              |      Agent (agent_y)           |
                                          |        (tun mode)              |              |        (tun mode)              |
                                          |                                |              |                                |
                                          |                                |              |                                |
                                          | +-----+tun interface+-------+  |              | +-----+tun interface+-------+  |
                                          |                                |              |                                |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          | |         IPv6+based         | |              | |         IPv6+based         | |
                                          | |        communicating       | |              | |        communicating       | |
                                          | |      piece of software     | |              | |      piece of software     | |
                                          | |      (e.g. coap client)    | |              | |      (e.g. coap server)    | |
                                          | |                            | |              | |                            | |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          |                                |              |                                |
                                          +--------------------------------+              +--------------------------------+
87
88
```

Federico Sismondi's avatar
Federico Sismondi committed
89
90
91
92
93
94
95
96
97
Event Bus API:
--------------

All the calls between the components are documented here:

[CORE API doc](http://doc.f-interop.eu/interop/)
and
[interop tests API doc](http://doc.f-interop.eu/interop/)

98
99


Federico Sismondi's avatar
Federico Sismondi committed
100
101
Running a test suite:
---------------------
Federico Sismondi's avatar
Federico Sismondi committed
102

Federico Sismondi's avatar
Federico Sismondi committed
103
user needs :
Federico Sismondi's avatar
Federico Sismondi committed
104

Federico Sismondi's avatar
Federico Sismondi committed
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
- an implementation under test (IUT) of a standard supported/protocol
by ioppytest framework, which either runs in some specific hardware or
locally in user's PC
- [the agent component](http://doc.f-interop.eu/interop/#agent)
which will route all the packets emitted from the IUT to the backend
and also to the second IUT (and viceversa)
- a user interface to help coordinating the tests
(either GUI or CLI component)

# Running a test suite using F-Interop platform

go to [go.f-interop.eu](go.f-interop.eu) and follow the instructions

# Running a test suite standalone

This mode of execution work for any of the following circustantces

- controls one IUT and wants to run tests against one of the
automated-IUTs the framework supports
- controls one IUT and is in direct contact with a second user
controlling a second IUT
- controls both implementations (IUTs) taking part in the interop
session

## Set up up the message broker

The interop testing tool use AMQP for sending messages between its
components, and the remote ones (like the agent). When running a
standalone setup the user first needs to have a RMQ broker running..

RMQ broker is a component which is **external** to the testing tool
and which establish the session and infrastructure so compomnents can
communicate with each other during the test session.
Federico Sismondi's avatar
Federico Sismondi committed
138

139
140
141
142
143
144
145
The options for this are:

- install locally RabbitMQ message broker on local machine,
create RMQ vhost, user, pass on local machine

    (# TODO add instructions)

Federico Sismondi's avatar
Federico Sismondi committed
146
147
148
149
150
- Request a remote vhost and credentials (user,pass) to
federico.sismondi@inria.fr (recommended)

don't hesitate to contact me, this is a very simple procedure and it's
free :D
151

Federico Sismondi's avatar
Federico Sismondi committed
152
## Export AMQP environment variables
153

154
155
156
after having a created vhost with its user/password,
export in the machine where the testing tool is running the following
env vars:
Federico Sismondi's avatar
Federico Sismondi committed
157

Federico Sismondi's avatar
Federico Sismondi committed
158
159
160
161
```
export AMQP_URL='amqp://someUser:somePassword@server/amqp_vhost'
export AMQP_EXCHANGE='amq.topic'
```
162

Federico Sismondi's avatar
Federico Sismondi committed
163
164
165
166
167
## Download the source code (see `Clonning the project` for more info)
```
git clone --recursive https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
cd ioppytest
```
168
169


Federico Sismondi's avatar
Federico Sismondi committed
170
## Build the testing tools
171

Federico Sismondi's avatar
Federico Sismondi committed
172
173
174
175
(docker, py2 and py3 needs to be installed in the machine)
```
make docker-build-all
```
176

Federico Sismondi's avatar
Federico Sismondi committed
177
178
179
180
## Run testing tool (CoAP testing tool example)
```
make run-coap-testing-tool
```
181

Federico Sismondi's avatar
Federico Sismondi committed
182
183
184
185
## Connect to the interop session using the CLI
```
make run-cli
```
186

Federico Sismondi's avatar
Federico Sismondi committed
187
## Connect the agent to the backend
188

Federico Sismondi's avatar
Federico Sismondi committed
189
if user's IUT is a CoAP client:
190

Federico Sismondi's avatar
Federico Sismondi committed
191
192
193
```
make run-agent-coap-client
```
194

Federico Sismondi's avatar
Federico Sismondi committed
195
if user's IUT is a CoAP server:
196

197
```
Federico Sismondi's avatar
Federico Sismondi committed
198
make run-agent-coap-server
199
200
```

Federico Sismondi's avatar
Federico Sismondi committed
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
## Running a second IUT

### User to user session, second user with his/her own IUT

The second IUT needs to connect to the same VHOST the same way first IUT
did. For this the RMQ broker needs to be reachable by this second IUT
 (and it's agent instance).

If this is the case then user 2 should just export the same environment
 variables as user 1, and launch agent, and CLI just as user 1 did.

### Single user session, agains an automated-IUT

 If the user wants to run test against one of the automated-IUT
 (~reference implementation) supported by ioppytest:
216

217
```
Federico Sismondi's avatar
Federico Sismondi committed
218
make run-coap-server
219
220
```

Federico Sismondi's avatar
Federico Sismondi committed
221
222
or for a coap client automated implementation:

223
```
Federico Sismondi's avatar
Federico Sismondi committed
224
make run-coap-client
225
226
```

Federico Sismondi's avatar
Federico Sismondi committed
227
228
229
230
231
232
233
234
235
236
237
238
239
240

## Running the interop session

Simply follow the CLI instructions and enjoy! :)


The implemented tests are based on this specification:
[ETSI CoAP test description](http://www.etsi.org/plugtests/CoAP/Document/CoAP_TestDescriptions_v015.pdf)


Developping a new test suite:
-----------------------------

## Clonning the project
241
```
Federico Sismondi's avatar
Federico Sismondi committed
242
243
git clone --recursive https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
cd ioppytest
244
245
```

Federico Sismondi's avatar
Federico Sismondi committed
246
### Attention with the git submodules!
247

Federico Sismondi's avatar
Federico Sismondi committed
248
remember when cloning a project with submodules to use --recursive flag
249
```
Federico Sismondi's avatar
Federico Sismondi committed
250
git clone --recursive ...
251
252
```

Federico Sismondi's avatar
Federico Sismondi committed
253
or else (in case you forgot about the flag), right after cloning you can:
254
```
Federico Sismondi's avatar
Federico Sismondi committed
255
git submodule update --init --recursive
256
257
```

Federico Sismondi's avatar
Federico Sismondi committed
258
259
260
261
262
263
whenever you find that your utils libraries are not the latests versions
you can 'bring' those last changes from the main utils repo to your project
with:
```
git submodule update --remote --merge
```
264

Federico Sismondi's avatar
Federico Sismondi committed
265
after bringing the last changes you can update your project with the last changes by doing:
266
```
Federico Sismondi's avatar
Federico Sismondi committed
267
268
269
git add <someSubModuleDir>
git commit -m 'updated submodule reference to last commit'
git push
270
271
```

Federico Sismondi's avatar
Federico Sismondi committed
272
273
274
275
276
277
278
279
280
281
282

## How to merge new features to upstream branch ?

Read CONTRIBUTING.rst document

## How can I develop & debug test suites?

for getting logs from the docker containers you can:
```
make get-logs
```
283

284
for running coap_testing_tool manually from docker api:
Federico Sismondi's avatar
Federico Sismondi committed
285

286
```
Federico Sismondi's avatar
Federico Sismondi committed
287
docker run -it
Federico Sismondi's avatar
Federico Sismondi committed
288
289
290
291
292
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
293
294
```

295
296
297

alternatively, if you are curious and you want to know
what's under the hood:
Federico Sismondi's avatar
Federico Sismondi committed
298

299
```
Federico Sismondi's avatar
Federico Sismondi committed
300
301
302
303
304
305
306
307
docker run -it
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
    bash

308
309
310
311
312
313
314
315
316
317
318
319
root@bab3b2220510:/coap_testing_tool# supervisord -c supervisor.conf
root@bab3b2220510:/coap_testing_tool# supervisorctl -c supervisor.conf
agent                            RUNNING   pid 28, uptime 0:00:02
automated-iut                    STARTING
bootstrap-agent-TT               RUNNING   pid 19, uptime 0:00:02
packet-router                    RUNNING   pid 24, uptime 0:00:02
packet-sniffer                   RUNNING   pid 18, uptime 0:00:02
tat                              RUNNING   pid 17, uptime 0:00:02
test-coordinator                 RUNNING   pid 26, uptime 0:00:02
supervisor>
```

Federico Sismondi's avatar
Federico Sismondi committed
320
321
or you can also run directly the processes without docker:
(supervisord needed)
322

Federico Sismondi's avatar
Federico Sismondi committed
323
324
325
326
```
sudo -E supervisord -c routeToConfigurationFileForTheTestSuite
sudo supervisorctl -c routeToConfigurationFileForTheTestSuite
```
327

Federico Sismondi's avatar
Federico Sismondi committed
328
329
note: use -E when launching supervisor process, it preserves the
env vars
330
331


Federico Sismondi's avatar
Federico Sismondi committed
332
333
FAQ
---
334

Federico Sismondi's avatar
Federico Sismondi committed
335
- How can I install docker on my machine?
336

Federico Sismondi's avatar
Federico Sismondi committed
337
    For this just follow this instructions: https://docs.docker.com/engine/installation/
338

Federico Sismondi's avatar
Federico Sismondi committed
339

Federico Sismondi's avatar
Federico Sismondi committed
340
341
342
343
- How do I install supervisord on my machine?
    Install supervisor (needed for spawning and monitoring processes):
    For this follow this instructions:
    http://supervisord.org/installing.html
Federico Sismondi's avatar
Federico Sismondi committed
344

Federico Sismondi's avatar
Federico Sismondi committed
345
346
- I have my own CoAP implementation, how can I add it as an
automated-IUT into CoAP Testing Tool:
Federico Sismondi's avatar
Federico Sismondi committed
347

Federico Sismondi's avatar
Federico Sismondi committed
348
    please contact federico.sismondi@inria.fr
349
350


Federico Sismondi's avatar
Federico Sismondi committed
351
- Docker build returns a "cannot fetch package" or a "cannot resolve .."
352

Federico Sismondi's avatar
Federico Sismondi committed
353
    try using ```--no-cache``` for the docker build
354

Federico Sismondi's avatar
Federico Sismondi committed
355
    more info http://stackoverflow.com/questions/24991136/docker-build-could-not-resolve-archive-ubuntu-com-apt-get-fails-to-install-a