README.md 13.4 KB
Newer Older
Federico Sismondi's avatar
Federico Sismondi committed
1
2
ioppytest framework:
--------------------
Federico Sismondi's avatar
Federico Sismondi committed
3

Federico Sismondi's avatar
Federico Sismondi committed
4
ioppytest is a framework for running interoperability tests.
Federico Sismondi's avatar
Federico Sismondi committed
5

Federico Sismondi's avatar
Federico Sismondi committed
6
7
8
9
10
11
12
13
14
15
This initial version tackles technical interoperability testing (CoAP
and 6LoWPAN interop tests), and effort is being made to implement
interop semantic tests notably for running tests in the WoT and OneM2M
context.

This repo contains all necessary software (and their dependencies) for
running a interoperability test sessions between two implementations
under test (IUT).

This can be run as standalone software and also integrated to f-interop
Federico Sismondi's avatar
Federico Sismondi committed
16
17
18
architecture.


Federico Sismondi's avatar
Federico Sismondi committed
19
20
Implemented test suites in the ioppytest framework:
---------------------------------------------------
Federico Sismondi's avatar
Federico Sismondi committed
21

Federico Sismondi's avatar
Federico Sismondi committed
22
23
The test suites implemented which are currently supported by the
framework are:
Federico Sismondi's avatar
Federico Sismondi committed
24

Federico Sismondi's avatar
Federico Sismondi committed
25
26
27
- CoAP Test Suite (user's IUT vs automated-IUT)
- CoAP Test Suite (between two users' IUT)
- 6LoWPAN Test Suite (between two users' IUT) (WIP)
Federico Sismondi's avatar
Federico Sismondi committed
28

29
30


Federico Sismondi's avatar
Federico Sismondi committed
31
32
Test setup:
-----------
Federico Sismondi's avatar
Federico Sismondi committed
33

Federico Sismondi's avatar
Federico Sismondi committed
34
35
36
An interop session happens between two implementation under test (IUT),
the following diagram shows the setup including the IUTs, testing tool
and auxiliary components needed for running a interop session.
37

Federico Sismondi's avatar
Federico Sismondi committed
38
39
All interactions between components take place using the AMQP event bus
(AMQP pub/sub mechanism)
40
41

```
Federico Sismondi's avatar
Federico Sismondi committed
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
                    +----------------------------+             +----------------------------+             +----------------------------+
                    |                            |             |                            |             |                            |
                    |    ioppytest Test Tool     |             |     User Interface         |             |     User Interface         |
                    |(CoAP, 6LoWPAN, OneM2M, etc)|             |         (user 1)           |             |         (user 2)           |
                    |                            |             |                            |             |                            |
                    |                            |             |                            |             |                            |
                    +----------------------------+             +----------------------------+             +----------------------------+

                             ^    +                                     ^    +                                     ^    +
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_x.tun.packet |    | toAgent.agent_y.tun.packet          |    |  ui.user1.step_verify.reply         |    |
                             |    |                                     |    |                                     |    |
fromAgent.agent_y.tun.packet |    | toAgent.agent_x.tun.packet          |    |                                     |    |
                             |    |                                     |    |                                     |    |
                             |    | ui.user1.step_verify.request        |    |                                     |    |
                             +    v                                     +    v                                     +    v

        +------------------------------------------------------------------------------------------------------------------------------------------------>
                                                                         AMQP Event Bus
        <-------------------------------------------------------------------------------------------------------------------------------------------------+

                                                       +     ^                                        +     ^
                                                       |     | toAgent.agent_x.tun.packetet           |     |  fromAgent.agent_y.tun.packet
                              data.tun.toAgent.agent_x |     |                                        |     |
                                                       |     |              toAgent.agent_y.tun.packet|     |
                                                       v     |                                        v     |
                                PC        +------------+-----+-------------+              +-----------+-----+--------------+
                                user 1    |                                |              |                                |
                                          |      Agent (agent_x)           |              |      Agent (agent_y)           |
                                          |        (tun mode)              |              |        (tun mode)              |
                                          |                                |              |                                |
                                          |                                |              |                                |
                                          | +-----+tun interface+-------+  |              | +-----+tun interface+-------+  |
                                          |                                |              |                                |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          | |         IPv6+based         | |              | |         IPv6+based         | |
                                          | |        communicating       | |              | |        communicating       | |
                                          | |      piece of software     | |              | |      piece of software     | |
                                          | |      (e.g. coap client)    | |              | |      (e.g. coap server)    | |
                                          | |                            | |              | |                            | |
                                          | +----------------------------+ |              | +----------------------------+ |
                                          |                                |              |                                |
                                          +--------------------------------+              +--------------------------------+
87
88
```

Federico Sismondi's avatar
Federico Sismondi committed
89
90
91
92
93
94
95
96
97
Event Bus API:
--------------

All the calls between the components are documented here:

[CORE API doc](http://doc.f-interop.eu/interop/)
and
[interop tests API doc](http://doc.f-interop.eu/interop/)

98
99


Federico Sismondi's avatar
Federico Sismondi committed
100
101
Running a test suite:
---------------------
Federico Sismondi's avatar
Federico Sismondi committed
102

Federico Sismondi's avatar
Federico Sismondi committed
103
user needs :
Federico Sismondi's avatar
Federico Sismondi committed
104

Federico Sismondi's avatar
Federico Sismondi committed
105
106
- an implementation under test (IUT) of a standard supported/protocol
by ioppytest framework, which either runs in some specific hardware or
Federico Sismondi's avatar
Federico Sismondi committed
107
locally in user's PC, e.g. a coap client implementation
108
-  run
Federico Sismondi's avatar
Federico Sismondi committed
109
[the agent component](http://doc.f-interop.eu/interop/#agent)
Federico Sismondi's avatar
Federico Sismondi committed
110
111
112
113
114
which will route all the packets emitted from the IUT to the backend
and also to the second IUT (and viceversa)
- a user interface to help coordinating the tests
(either GUI or CLI component)

115
For simplifying the access to CLI, agent and other components, ioppytest
116
117
118
inlcudes a Makefile, with it you can use `make cmd`,
for more information execute `make help`

Federico Sismondi's avatar
Federico Sismondi committed
119
120
121
122
123
124
# Running a test suite using F-Interop platform

go to [go.f-interop.eu](go.f-interop.eu) and follow the instructions

# Running a test suite standalone

Federico Sismondi's avatar
Federico Sismondi committed
125
This mode of execution work for any of the following circumstances
Federico Sismondi's avatar
Federico Sismondi committed
126

Federico Sismondi's avatar
Federico Sismondi committed
127
- user controls one IUT and wants to run tests against one of the
Federico Sismondi's avatar
Federico Sismondi committed
128
automated-IUTs the framework supports
Federico Sismondi's avatar
Federico Sismondi committed
129
- user controls one IUT and is in direct contact with a second user
Federico Sismondi's avatar
Federico Sismondi committed
130
controlling a second IUT
Federico Sismondi's avatar
Federico Sismondi committed
131
- user controls both implementations (IUTs) taking part in the interop
Federico Sismondi's avatar
Federico Sismondi committed
132
133
134
135
136
137
138
139
140
141
142
session

## Set up up the message broker

The interop testing tool use AMQP for sending messages between its
components, and the remote ones (like the agent). When running a
standalone setup the user first needs to have a RMQ broker running..

RMQ broker is a component which is **external** to the testing tool
and which establish the session and infrastructure so compomnents can
communicate with each other during the test session.
Federico Sismondi's avatar
Federico Sismondi committed
143

144
145
146
147
148
149
150
The options for this are:

- install locally RabbitMQ message broker on local machine,
create RMQ vhost, user, pass on local machine

    (# TODO add instructions)

Federico Sismondi's avatar
Federico Sismondi committed
151
152
153
154
155
- Request a remote vhost and credentials (user,pass) to
federico.sismondi@inria.fr (recommended)

don't hesitate to contact me, this is a very simple procedure and it's
free :D
156

Federico Sismondi's avatar
Federico Sismondi committed
157
## Export AMQP environment variables
158

159
160
161
after having a created vhost with its user/password,
export in the machine where the testing tool is running the following
env vars:
Federico Sismondi's avatar
Federico Sismondi committed
162

Federico Sismondi's avatar
Federico Sismondi committed
163
164
165
166
```
export AMQP_URL='amqp://someUser:somePassword@server/amqp_vhost'
export AMQP_EXCHANGE='amq.topic'
```
167

Federico Sismondi's avatar
Federico Sismondi committed
168
169
170
171
172
## Download the source code (see `Clonning the project` for more info)
```
git clone --recursive https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
cd ioppytest
```
173
174


Federico Sismondi's avatar
Federico Sismondi committed
175
## Build the testing tools
176

Federico Sismondi's avatar
Federico Sismondi committed
177
178
(docker, py2 and py3 needs to be installed in the machine)
```
179
make build-all
Federico Sismondi's avatar
Federico Sismondi committed
180
```
181

Federico Sismondi's avatar
Federico Sismondi committed
182
183
184
185
## Run testing tool (CoAP testing tool example)
```
make run-coap-testing-tool
```
186

Federico Sismondi's avatar
Federico Sismondi committed
187
188
189
190
## Connect to the interop session using the CLI
```
make run-cli
```
191

Federico Sismondi's avatar
Federico Sismondi committed
192
## Connect the agent to the backend
193

Federico Sismondi's avatar
Federico Sismondi committed
194
if user's IUT is a CoAP client:
195

Federico Sismondi's avatar
Federico Sismondi committed
196
197
198
```
make run-agent-coap-client
```
199

Federico Sismondi's avatar
Federico Sismondi committed
200
if user's IUT is a CoAP server:
201

202
```
Federico Sismondi's avatar
Federico Sismondi committed
203
make run-agent-coap-server
204
205
```

Federico Sismondi's avatar
Federico Sismondi committed
206
207
208
209
210
211
212
213
214
215
216
## Running a second IUT

### User to user session, second user with his/her own IUT

The second IUT needs to connect to the same VHOST the same way first IUT
did. For this the RMQ broker needs to be reachable by this second IUT
 (and it's agent instance).

If this is the case then user 2 should just export the same environment
 variables as user 1, and launch agent, and CLI just as user 1 did.

217
### Single user session, against an automated-IUT
Federico Sismondi's avatar
Federico Sismondi committed
218
219
220

 If the user wants to run test against one of the automated-IUT
 (~reference implementation) supported by ioppytest:
221

222
```
Federico Sismondi's avatar
Federico Sismondi committed
223
make run-coap-server
224
225
```

Federico Sismondi's avatar
Federico Sismondi committed
226
227
or for a coap client automated implementation:

228
```
Federico Sismondi's avatar
Federico Sismondi committed
229
make run-coap-client
230
231
```

Federico Sismondi's avatar
Federico Sismondi committed
232
233
234
235
236
237
238
239
240
241
242
243
244
245

## Running the interop session

Simply follow the CLI instructions and enjoy! :)


The implemented tests are based on this specification:
[ETSI CoAP test description](http://www.etsi.org/plugtests/CoAP/Document/CoAP_TestDescriptions_v015.pdf)


Developping a new test suite:
-----------------------------

## Clonning the project
246
```
Federico Sismondi's avatar
Federico Sismondi committed
247
248
git clone --recursive https://gitlab.f-interop.eu/f-interop-contributors/ioppytest.git
cd ioppytest
249
250
```

Federico Sismondi's avatar
Federico Sismondi committed
251
### Attention with the git submodules!
252

Federico Sismondi's avatar
Federico Sismondi committed
253
remember when cloning a project with submodules to use --recursive flag
254
```
Federico Sismondi's avatar
Federico Sismondi committed
255
git clone --recursive ...
256
257
```

Federico Sismondi's avatar
Federico Sismondi committed
258
or else (in case you forgot about the flag), right after cloning you can:
259
```
Federico Sismondi's avatar
Federico Sismondi committed
260
git submodule update --init --recursive
261
262
```

Federico Sismondi's avatar
Federico Sismondi committed
263
264
265
266
267
268
whenever you find that your utils libraries are not the latests versions
you can 'bring' those last changes from the main utils repo to your project
with:
```
git submodule update --remote --merge
```
269

Federico Sismondi's avatar
Federico Sismondi committed
270
after bringing the last changes you can update your project with the last changes by doing:
271
```
Federico Sismondi's avatar
Federico Sismondi committed
272
273
274
git add <someSubModuleDir>
git commit -m 'updated submodule reference to last commit'
git push
275
276
```

Federico Sismondi's avatar
Federico Sismondi committed
277
278
279
280
281
282
283
284
285
286
287

## How to merge new features to upstream branch ?

Read CONTRIBUTING.rst document

## How can I develop & debug test suites?

for getting logs from the docker containers you can:
```
make get-logs
```
288

289
for running coap_testing_tool manually from docker api:
Federico Sismondi's avatar
Federico Sismondi committed
290

291
```
Federico Sismondi's avatar
Federico Sismondi committed
292
docker run -it
Federico Sismondi's avatar
Federico Sismondi committed
293
294
295
296
297
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
298
299
```

300
301
302

alternatively, if you are curious and you want to know
what's under the hood:
Federico Sismondi's avatar
Federico Sismondi committed
303

304
```
Federico Sismondi's avatar
Federico Sismondi committed
305
306
307
308
309
310
311
312
docker run -it
    --env AMQP_EXCHANGE=$AMQP_EXCHANGE
    --env AMQP_URL=$AMQP_URL
    --sysctl net.ipv6.conf.all.disable_ipv6=0
    --privileged
    testing_tool-interoperability-coap
    bash

313
314
root@bab3b2220510:/ioppytest# supervisord -c supervisor.conf
root@bab3b2220510:/ioppytest# supervisorctl -c supervisor.conf
315
316
317
318
319
320
321
322
agent                            RUNNING   pid 28, uptime 0:00:02
packet-router                    RUNNING   pid 24, uptime 0:00:02
packet-sniffer                   RUNNING   pid 18, uptime 0:00:02
tat                              RUNNING   pid 17, uptime 0:00:02
test-coordinator                 RUNNING   pid 26, uptime 0:00:02
supervisor>
```

Federico Sismondi's avatar
Federico Sismondi committed
323
324
or you can also run directly the processes without docker:
(supervisord needed)
325

Federico Sismondi's avatar
Federico Sismondi committed
326
327
```
sudo -E supervisord -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
328
sudo -E supervisorctl -c routeToConfigurationFileForTheTestSuite
Federico Sismondi's avatar
Federico Sismondi committed
329
```
330

Federico Sismondi's avatar
Federico Sismondi committed
331
332
333
you can use for example envs/coap_testing_tool/supervisor.conf.ini
for using the coap_testing_tool

Federico Sismondi's avatar
Federico Sismondi committed
334
335
note: use -E when launching supervisor process, it preserves the
env vars
336
337


Federico Sismondi's avatar
Federico Sismondi committed
338
339
FAQ
---
340

Federico Sismondi's avatar
Federico Sismondi committed
341
- How can I install docker on my machine?
342

Federico Sismondi's avatar
Federico Sismondi committed
343
    For this just follow this instructions: https://docs.docker.com/engine/installation/
344

Federico Sismondi's avatar
Federico Sismondi committed
345

Federico Sismondi's avatar
Federico Sismondi committed
346
347
348
349
- How do I install supervisord on my machine?
    Install supervisor (needed for spawning and monitoring processes):
    For this follow this instructions:
    http://supervisord.org/installing.html
Federico Sismondi's avatar
Federico Sismondi committed
350

Federico Sismondi's avatar
Federico Sismondi committed
351
352
- I have my own CoAP implementation, how can I add it as an
automated-IUT into CoAP Testing Tool:
Federico Sismondi's avatar
Federico Sismondi committed
353

Federico Sismondi's avatar
Federico Sismondi committed
354
    please contact federico.sismondi@inria.fr
355
356


Federico Sismondi's avatar
Federico Sismondi committed
357
- Docker build returns a "cannot fetch package" or a "cannot resolve .."
358

Federico Sismondi's avatar
Federico Sismondi committed
359
    try using ```--no-cache``` for the docker build
360

Federico Sismondi's avatar
Federico Sismondi committed
361
    more info http://stackoverflow.com/questions/24991136/docker-build-could-not-resolve-archive-ubuntu-com-apt-get-fails-to-install-a