hpcwiki - New pages [en]

MMS: M&S Principles 2018 -II

2018-11-16T11:57:27Z

Hfrancot: Created page with "* Epistemología del M&S File:EpistemologiaMS.pdf * Modelado de sistemas mediante ecuaciones diferenciales en derivadas parciales File:PDEs.pdf"

* Epistemología del M&S [[File:EpistemologiaMS.pdf]]
* Modelado de sistemas mediante ecuaciones diferenciales en derivadas parciales [[File:PDEs.pdf]]

EngPracII: TutorialOverleaf

2018-07-31T15:28:53Z

Gsosa: /* Plantilla de documento escrito UCENTRAL */

== Plantilla de documento escrito UCENTRAL ==
Descarga la plantilla en Overleaf del documento de Práctica de Ingeniería II haciendo click [https://www.overleaf.com/read/rvyfffdhczmh '''aquí''']

== Previsualización de documento terminado ==

[[File:EjemploDocOverleaf.png|1000 px]]

UNED: Plan de trabajo - Posturografía

2018-07-27T10:03:31Z

Hfrancot: /* Estabilograma */

[[http://hpclab.ucentral.edu.co/wiki/index.php/Collaborations#UNED Back]]
===Estabilograma===
* Tesis de Germán Sosa
** artículo
** revisar estadística
* Movimiento browniano/no browniano
** se adecua al modelo?
** formas de caracterización: marcadores biomecánicos
** test: calcular la segunda derivada para buscar aceleraciones impulsivas
*** Modelo mecánico del estabilograma

UNED: Plan de trabajo - Clima

2018-07-27T09:59:13Z

Daniel: /* Clima */

[[http://hpclab.ucentral.edu.co/wiki/index.php/Collaborations#UNED Back]]

=== Clima ===
* Revisar la tesis de Juan Felipe Molina
* Definir taxonomía de estructuras (Sandra Mejía)
** Establecer caracterización (1)
* Deep learning: campo de temperatura (en función de P) vs. velocidad (en función de P)
** Remapear datos en P
** Taxonomía desde imágenes de satélite (?)
*** Desde las imágenes se puede calcular la temperatura en función de P
** Repositorio NOAA de imágenes de satélite?

* Detección y extracción automática de patrones (2)
** relacionar estructuras con eventos extremos (e.g. incendios)

UNED: Plan de trabajo

2018-07-27T09:56:04Z

Hfrancot: Created page with "===Clima === - revisar la tesis de Juan Felipe - taxonomía de estructuras * caracterización (1) - Deep learning: campo de temperatura (en función de P) vs. velocidad (en f..."

===Clima ===
- revisar la tesis de Juan Felipe
- taxonomía de estructuras
* caracterización (1)
- Deep learning: campo de temperatura (en función de P) vs. velocidad (en función de P)
* remapear datos en P
* taxonomía desde imágenes de satélite?
+ desde las imágenes se puede calcular la temperatura en función de P
* repositorio NOAA de imágenes de satélite?
- Detección y extracción automática de patrones (2)
* relacionar estructuras con eventos extremos (e.g. incendios)

HPC:NFS configuration

2018-04-22T03:29:51Z

Hfrancot: /* Raw guide contents */

=== NFS configuration to provide HPCLab DAS access to worker nodes ===
* This guide was followed OK
** https://www.howtoforge.com/nfs-server-and-client-on-centos-7
* Node 11 needs nfs service permanent public access revoke

==== Raw guide contents ====
1 Preliminary Note
I have fresh installed CentOS 7 server, on which I am going to install the NFS server. My CentOS server have hostname server1.example.com and IP as 192.168.0.100

If you don't have a CentOS server installed yet, use this tutorial for the basic operating system installation. Additionally to the server, we need a CentOS 7 client machine, this can be either a server or desktop system. In my case, I will use a CentOS 7 desktop with hostname client1.example.com and IP 192.168.0.101 as client. I will run all the commands in this tutorial as root user.

2 At NFS server end
As the first step, we will install these packages on the CentOS server with yum:

yum install nfs-utils

Now create the directory that will be shared by NFS:

mkdir /var/nfsshare

Change the permissions of the folder as follows:

chmod -R 755 /var/nfsshare
chown nfsnobody:nfsnobody /var/nfsshare

We use /var/nfsshare as shared folder, if we use another drive such as the /home directory, then the permission chnges will cause a massive permissions problem and ruin the whole hierarchy. So in case we want to share the /home directory then permissions must not be changed.
Next, we need to start the services and enable them to be started at boot time.

systemctl enable rpcbind
systemctl enable nfs-server
systemctl enable nfs-lock
systemctl enable nfs-idmap
systemctl start rpcbind
systemctl start nfs-server
systemctl start nfs-lock
systemctl start nfs-idmap

Now we will share the NFS directory over the network a follows:

nano /etc/exports

We will make two sharing points /home and /var/nfsshare. Edit the exports file as follows:

/var/nfsshare 192.168.0.101(rw,sync,no_root_squash,no_all_squash)
/home 192.168.0.101(rw,sync,no_root_squash,no_all_squash)

Note 192.168.0.101 is the IP of client machine, if you wish that any other client should access it you need to add the it IP wise other wise you can add "*" instead of IP for all IP access.

Condition is that it must be pingable at both ends.

Finally, start the NFS service:

systemctl restart nfs-server

Again we need to add the NFS service override in CentOS 7 firewall-cmd public zone service as:

firewall-cmd --permanent --zone=public --add-service=nfs
firewall-cmd --permanent --zone=public --add-service=mountd
firewall-cmd --permanent --zone=public --add-service=rpc-bind
firewall-cmd --reload

Note: If it will be not done, then it will give error for Connection Time Out at client side.

Now we are ready with the NFS server part.

3 NFS client end
In my case, I have a CentOS 7 desktop as client. Other CentOS versions will also work the same way. Install the nfs-utild package as follows:

yum install nfs-utils

Now create the NFS directory mount points:

mkdir -p /mnt/nfs/home
mkdir -p /mnt/nfs/var/nfsshare

Next, we will mount the NFS shared home directory in the client machine as shown below:

mount -t nfs 192.168.0.100:/home /mnt/nfs/home/

It will mount /home of NFS server. Next we will mount the /var/nfsshare directory:

mount -t nfs 192.168.0.100:/var/nfsshare /mnt/nfs/var/nfsshare/

Now we are connected with the NFS share, we will crosscheck it as follows:

df -kh

[root@client1 ~]# df -kh

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 39G 1.1G 38G 3% /
devtmpfs 488M 0 488M 0% /dev
tmpfs 494M 0 494M 0% /dev/shm
tmpfs 494M 6.7M 487M 2% /run
tmpfs 494M 0 494M 0% /sys/fs/cgroup
/dev/mapper/centos-home 19G 33M 19G 1% /home
/dev/sda1 497M 126M 372M 26% /boot
192.168.0.100:/var/nfsshare 39G 980M 38G 3% /mnt/nfs/var/nfsshare
192.168.0.100:/home 19G 33M 19G 1% /mnt/nfs/home
[root@client1 ~]#

So we are connected with the NFS share.

Now we will check the read/write permissions in the shared path. At client enter the command:

touch /mnt/nfs/var/nfsshare/test_nfs

So we successfully configured an NFS-share.

4 Permanent NFS mounting
We have to re-mount the NFS share at the client after every reboot. Here are the steps to mount it permanently by adding the NFS-share in /etc/fstab file of client machine:

nano /etc/fstab

Add the entries like this:

[...]
192.168.0.100:/home /mnt/nfs/home nfs defaults 0 0
192.168.0.100:/var/nfsshare /mnt/nfs/var/nfsshare nfs defaults 0 0

Note 192.168.0.100 is the server NFS-share IP address, it will vary in your case.

This will make the permanent mount of the NFS-share. Now you can reboot the machine and mount points will be permanent even after the reboot.

Cheers, now we have a successfully configured NFS-server over CentOS 7 :)

5 Links
CentOS : http://www.centos.org/

HPCLab:Specific PostgreSQL configuration

2018-04-22T01:32:21Z

Hfrancot: /* PostgreSQL 9.4 */

=== PostgreSQL 9.4 ===

* Main guide: http://www.sfentona.net/?p=2756
** Update: CentOS version is now pgdg-centos94-9.4-3.noarch.rpm (correct the download)
* The initdb process (create the pgdata structure in the file system) yielded:
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

./postgres -D /media/hpclab-data/pgdata
or
./pg_ctl -D /media/hpclab-data/pgdata -l logfile start

=== Another cool guide ===
* https://www.digitalocean.com/community/tutorials/how-to-move-a-postgresql-data-directory-to-a-new-location-on-ubuntu-16-04

=== Complete "raw" guide ===
Step 1) Install the yum Repo for PostgresSQL 9.4

yum -y localinstall http://yum.postgresql.org/9.4/redhat/rhel-7-x86_64/pgdg-centos94-9.4-1.noarch.rpm

Step 2) Download the RPMS and install the database engine

yum -y install postgresql94-contrib postgresql94-server

Step 3) Create the directories from where the database will be running from. In this case it will be from /home/postgres

mkdir /home/postgres; chown -R postgres:postgres /home/postgres
Step 4) Initialize the database.

cd /usr/pgsql-9.4/bin/; ./initdb -D /home/postgres

Step 5) Alter the default location from where your database will be running from.

vim /var/lib/pgsql/.bash_profile

#Alter the PGDATA as follows. See sample below
#PGDATA=/var/lib/pgsql/9.4/data
PGDATA=/home/postgres
export PGDATA

6) Configure systemd to start the database on boot-up

vi /usr/lib/systemd/system/postgresql-9.4.service

# Note: avoid inserting whitespace in these Environment= lines, or you may
# break postgresql-setup.
# Location of database directory
#Environment=PGDATA=/var/lib/pgsql/9.4/data/
Environment=PGDATA=/home/postgres/

Run as superuser

systemctl daemon-reload
systemctl enable postgresql-9.4.service
systemctl start postgresql-9.4.service

HPC:Tensorflow

2018-04-21T23:31:57Z

Hfrancot: /* Functionality test */

=== Basic Anaconda Tensorflow GPU installation ===

By April 21st, 2018, Anaconda 3 (i.e. running Python 3.5 or above) has evolved enough to provide functional GPU-based instances of Tensorflow, running on iPython or Jupyter notebook.

In the worker nodes, it is recommended to use non interactive scripts; however, it is under study the provision of Jupyter-notebook and/or iPython in the worker nodes (not recommended). A better alternative to be addressed is the installation of Jupyter-hub in the frontend to control python slave processes in the worker nodes.

To test the GPU-based Tensorflow functionality on the worker nodes, the following procedure has been performed

* Create a test user:
# adduser test
# passwd test
# su test
$ cd ~

* Download and install Anaconda3 v. 5.1.0
$ wget https://repo.anaconda.com/archive/Anaconda3-5.1.0-Linux-x86_64.sh
$ bash Anaconda3-5.1.0-Linux-x86_64.sh
$ cd anaconda3/bin
$ ./conda install tensorflow-gpu

=== Functionality test ===

* Test in iPython the GPU functionality and Tensorflow general behavior
$ ./ipython

* Code for iPython:

'''import''' tensorflow '''as''' tf
''# Creates a graph.''
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
''# Creates a session with log_device_placement set to True.''
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
''# Runs the op.''
print(sess.run(c))

* Test output should looks like (alongside warnings and other stuff)
Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: Tesla K40c, pci bus
id: 0000:05:00.0
b: /job:localhost/replica:0/task:0/device:GPU:0
a: /job:localhost/replica:0/task:0/device:GPU:0
MatMul: /job:localhost/replica:0/task:0/device:GPU:0
[[ 22. 28.]
[ 49. 64.]]

Data Analysis Problem definition

2018-04-03T22:54:10Z

Hfrancot: Created page with "=== Data Analysis Problem Definition === * Project title: * Student: * Research question/Problem formulation: * Main objective: * Target variables: ** Variable description: *..."

=== Data Analysis Problem Definition ===

* Project title:
* Student:
* Research question/Problem formulation:
* Main objective:
* Target variables:
** Variable description:
** Data types:
* Dataset description:
** Data sources:
** Number of records:
** Original format:
** Proposed data representation:
* Suggested method (general description):
** Project stages/work packages:
** Required computational resources:
* Expected results and contribution:
* Suggested baseline works/references:

EngPracII: Systematic Review

2018-03-06T15:55:15Z

Gsosa:

Pasos para hacer una selección sistemática de trabajos previos para la consolidación del estado del arte

#Identificación
#Filtrado (Screening)
#Elegibilidad
#Trabajos incluidos

[[File:systematicReview.png|600px]]

EngPracII: Estructuración del problema

2018-02-27T15:58:33Z

Gsosa:

El Documento final de Práctica II debe presentar la estructura de secciones presentada a continuación:

#Título
#Introducción
#Contextualización del problema 
## Contextualización del problema
## Contextualización metodológica
#Objetivo
#Estado del arte
##Diagrama PRISMA
##* [[File:systematicReview.png|300px]]
##Resumen de propuestas, técnicas, tecnologías e implementaciones de soluciones 
##Resumen cuantitativo (meta-analísis)
##*[[File:meta_analysis.png|600px]]
#Conclusiones
#Bibliografía

== Ejemplos ==
[[File:Revisión_en_análisis_automático_de_sonidos_pulmonares.pdf]]

== Guías ==
[[http://www.colombiaaprende.edu.co/html/investigadores/1609/articles-322806_recurso_1.pdf Guía para construir un estado del arte - Colombia Aprende]]

MMS: Data Mining (2018-I)

2018-02-06T22:49:40Z

Hfrancot: /* Session 5 */

== Course contents ==

=== Chapter 1. Introduction to PYTHON ===

==== Session 1 ====
* Downloading and installing PYTHON
* Installing and loading packages
* Reading and writing data
* Converting types on character variables

# Presentation [[File:MMS DataMining1.pdf| slides]]
## First script [http://hpclab.ucentral.edu.co/~hfranco/session1_first_script.ipynb]

=== Chapter 2. Data Exploration with Real Live Data ===
==== Session 2 ====
* Using PYTHON to manipulate data
* Reading a dataset from a CSV file
* Applying basic statistics
* Working with univariate descriptive statistics in PYTHON
* Performing correlations.
* Operating a probability distribution in PYTHON
* Fitting a linear regression model with Python
* Summarizing linear model fits

'''Files for Download:'''
* Session 2 (iPython Notebook) [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/2ndSessionPython-13-02-2018.ipynb]
* Data examples:
** Text File [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/animals.txt]
** Excel File [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/93CAR.xlsx]
** CSV File [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/Yayacon.csv]
* Interactive Regression - Machine Learning (iPython Notebook) [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/InteractiveRegressionML.ipynb]
* Interactive Logistic Regression (iPython Notebook) [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/InteractiveLogisticRegression.ipynb]
* Curve Fitting (iPython Notebook) [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/Ajuste_Curvas_1.ipynb]
'''Homework - Workshop 1:'''
* Workshop iPython Notebook: [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/TallerNo1.ipynb]
* Dataset 1 (mobile phone robbering data sample -Colombia, 2017): [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/muestra_hurto.xlsx]
* Dataset 2 (customer database -marketing): [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/customer_dbase.xlsx]

==== Session 3 ====
* Detecting missing values
* Imputing missing values
* Understanding data sampling in PYTHON
* Parametric and non-parametric statistical inference
* Conducting an exact binomial test

'''Files to download'''

* Sampling and imputation [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/3rdSession-FaltantesMuestreo.ipynb]
* Normal Distribution [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/3rdSession-NormalDistribution.ipynb]
* Example data [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/INMISIONES.xlsx]
'''Homework - Workshop 2:'''
* Workshop iPython Notebook: [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/Taller2_Data_Mining_MMS.ipynb]

==== Session 4 ====
* Performing the Kolmogorov-Smirnov test
* Understanding the Wilcoxon Rank Sum and Signed Rank test
* Working with Pearson’s Chi-squared test
* Conducting a one-way ANOVA
* Performing a two-way ANOVA
'''Files'''
* Nonparametric tests [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/4thSession-NoParametricas.ipynb]
* Example data [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/bank-additional-full.xlsx]

==== Special Session: Project definition ====
* Propose the course final project based on a real-life challenging data analysis problem
* Form: [[Data Analysis Problem definition]]

==== Session 5 ====
* Introduction to Machine Learning
* Supervised and unsupervised learning
* Getting a suitable dataset
* Predicting passenger survival with a decision tree
* Validating the power of prediction with a confusion matrix
* Assessing performance with the ROC curve

'''Files:'''
* Session presentation [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/SessionV/VSession.pdf]
* Session notebook [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/SessionV/V_Session_10-04-2018.ipynb]
* Balance scale data file [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/SessionV/balance-scale.data]
* Titanic test [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/SessionV/titanic_test.xlsx]
* Titanic test [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/SessionV/titanic_train.xlsx]
* Graphviz [http://hpclab.ucentral.edu.co/~hfranco/data_analysis/SessionV/graphviz-2.38.zip]

=== Chapter 2. Classification (I) – Tree, Lazy, and Probabilistic ===

==== Session 6 ====
* Preparing the training and testing datasets
* Building a classification model with recursive partitioning trees
* Visualizing a recursive partitioning tree
* Measuring the prediction performance of a recursive partitioning tree
* Pruning a recursive partitioning tree

==== Session 7 ====

* Building a classification model with a conditional inference tree
* Visualizing a conditional inference tree
* Measuring the prediction performance of a conditional inference tree
* Classifying data with logistic regression

==== Applied Material (to be updated) ====

=== Chapter 3. Working with Spatial Data ===

==== Session 8 ====

* Motivation: What's so great about spatial data?
* Spatial data structures.
* Making maps.
** Static maps with Python.
** Projections.
** Geocoding, routes, and distances.
** Dynamic maps with leaflet.
* Extended example: Congressional districts.
** Election results.
** Congressional districts.
** Putting it all together.
** Using leaflet.
* Effective maps: How (not) to lie.
* Extended example: Historical airline route maps.
* Projecting polygons.