You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
simulacion-permeabilidad/fftma_module/gen/analysis.ipynb

1523 lines
49 KiB
Plaintext

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Análisis de la etapa de generación de medios"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Armado del dataset\n",
"\n",
"En este paso parsearemos los archivos para obtener estadísticas sobre el tiempo que tarda cada ejecución de una función, sobre la memoria usada, el uso de CPU (TODO). Con esto buscamos identificar:\n",
"- Qué funciones son las que consumen mayor cantidad de memoria\n",
"- Qué funciones son las que tienen un mayor tiempo de procesamiento\n",
"- Qué funciones son las que son invocadas una mayor cantidad de veces\n",
"\n",
"Una vez identificados estos puntos de análisis podemos proponer soluciones para mejorar estas estadísticas."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"def get_function_name(function_name):\n",
" return function_name[10:].rsplit(\".c\")[0]"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"relations = {\n",
" \"generate\": [\"gasdev\"],\n",
" \"fftma2\": [\"covariance\", \"fourt\", \"prebuild_gwn\"]\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def get_data(file_name):\n",
" data = []\n",
"\n",
" with open(file_name) as log_file:\n",
" lines = log_file.readlines()\n",
" for line in lines:\n",
" row = {}\n",
" split_line = line.split()\n",
" if \"USED\" not in split_line or \"ELAPSED\" not in split_line: continue\n",
" idx_used_mem = split_line.index(\"USED\") + 4\n",
" idx_elapsed = split_line.index(\"ELAPSED\") + 2\n",
" \n",
" function_name = get_function_name(split_line[2])\n",
" used_virtual_mem = float(split_line[idx_used_mem])\n",
" elapsed = float(split_line[idx_elapsed].rsplit(\",\")[0])\n",
"\n",
" # TODO: add CPU\n",
" row[\"function\"] = function_name\n",
" row[\"memory\"] = used_virtual_mem \n",
" row[\"time\"] = elapsed\n",
" data.append(row)\n",
" \n",
" return data"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"def create_df(file_name):\n",
" data = get_data(file_name)\n",
" df = pd.DataFrame(data)\n",
" return df.groupby(['function']).agg({'time': ['min', 'max', 'mean', 'sum', 'count'], 'memory': ['min', 'max', 'median']})"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"def analyze(file_name):\n",
" df_grouped = create_df(file_name)\n",
" return df_grouped.sort_values(by=('time', 'sum'), ascending=False) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## N = 8"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th></th>\n",
" <th colspan=\"3\" halign=\"left\">memory</th>\n",
" <th colspan=\"5\" halign=\"left\">time</th>\n",
" </tr>\n",
" <tr>\n",
" <th></th>\n",
" <th>min</th>\n",
" <th>max</th>\n",
" <th>median</th>\n",
" <th>min</th>\n",
" <th>max</th>\n",
" <th>mean</th>\n",
" <th>sum</th>\n",
" <th>count</th>\n",
" </tr>\n",
" <tr>\n",
" <th>function</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Py_kgeneration</th>\n",
" <td>2.9</td>\n",
" <td>2.9</td>\n",
" <td>2.9</td>\n",
" <td>0.091805</td>\n",
" <td>0.091805</td>\n",
" <td>0.091805</td>\n",
" <td>0.091805</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>generate</th>\n",
" <td>2.9</td>\n",
" <td>2.9</td>\n",
" <td>2.9</td>\n",
" <td>0.081707</td>\n",
" <td>0.081707</td>\n",
" <td>0.081707</td>\n",
" <td>0.081707</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>gasdev</th>\n",
" <td>0.0</td>\n",
" <td>0.5</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.005674</td>\n",
" <td>0.000093</td>\n",
" <td>0.047828</td>\n",
" <td>512</td>\n",
" </tr>\n",
" <tr>\n",
" <th>fftma2</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.007942</td>\n",
" <td>0.007942</td>\n",
" <td>0.007942</td>\n",
" <td>0.007942</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>covariance</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.007492</td>\n",
" <td>0.007492</td>\n",
" <td>0.007492</td>\n",
" <td>0.007492</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>ran2</th>\n",
" <td>0.0</td>\n",
" <td>0.5</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000018</td>\n",
" <td>0.000002</td>\n",
" <td>0.001708</td>\n",
" <td>702</td>\n",
" </tr>\n",
" <tr>\n",
" <th>cov_value</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000030</td>\n",
" <td>0.000001</td>\n",
" <td>0.000707</td>\n",
" <td>700</td>\n",
" </tr>\n",
" <tr>\n",
" <th>fourt</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000079</td>\n",
" <td>0.000107</td>\n",
" <td>0.000091</td>\n",
" <td>0.000274</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>cgrid</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000067</td>\n",
" <td>0.000067</td>\n",
" <td>0.000067</td>\n",
" <td>0.000067</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>length</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000008</td>\n",
" <td>0.000008</td>\n",
" <td>0.000008</td>\n",
" <td>0.000024</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>build_real</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000009</td>\n",
" <td>0.000009</td>\n",
" <td>0.000009</td>\n",
" <td>0.000009</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>prebuild_gwn</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000007</td>\n",
" <td>0.000007</td>\n",
" <td>0.000007</td>\n",
" <td>0.000007</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>maxfactor</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000001</td>\n",
" <td>0.000001</td>\n",
" <td>0.000001</td>\n",
" <td>0.000003</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>clean_real</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000002</td>\n",
" <td>0.000002</td>\n",
" <td>0.000002</td>\n",
" <td>0.000002</td>\n",
" <td>1</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" memory time \\\n",
" min max median min max mean sum \n",
"function \n",
"Py_kgeneration 2.9 2.9 2.9 0.091805 0.091805 0.091805 0.091805 \n",
"generate 2.9 2.9 2.9 0.081707 0.081707 0.081707 0.081707 \n",
"gasdev 0.0 0.5 0.0 0.000000 0.005674 0.000093 0.047828 \n",
"fftma2 0.0 0.0 0.0 0.007942 0.007942 0.007942 0.007942 \n",
"covariance 0.0 0.0 0.0 0.007492 0.007492 0.007492 0.007492 \n",
"ran2 0.0 0.5 0.0 0.000000 0.000018 0.000002 0.001708 \n",
"cov_value 0.0 0.0 0.0 0.000000 0.000030 0.000001 0.000707 \n",
"fourt 0.0 0.0 0.0 0.000079 0.000107 0.000091 0.000274 \n",
"cgrid 0.0 0.0 0.0 0.000067 0.000067 0.000067 0.000067 \n",
"length 0.0 0.0 0.0 0.000008 0.000008 0.000008 0.000024 \n",
"build_real 0.0 0.0 0.0 0.000009 0.000009 0.000009 0.000009 \n",
"prebuild_gwn 0.0 0.0 0.0 0.000007 0.000007 0.000007 0.000007 \n",
"maxfactor 0.0 0.0 0.0 0.000001 0.000001 0.000001 0.000003 \n",
"clean_real 0.0 0.0 0.0 0.000002 0.000002 0.000002 0.000002 \n",
"\n",
" \n",
" count \n",
"function \n",
"Py_kgeneration 1 \n",
"generate 1 \n",
"gasdev 512 \n",
"fftma2 1 \n",
"covariance 1 \n",
"ran2 702 \n",
"cov_value 700 \n",
"fourt 3 \n",
"cgrid 1 \n",
"length 3 \n",
"build_real 1 \n",
"prebuild_gwn 1 \n",
"maxfactor 3 \n",
"clean_real 1 "
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"analyze('log_8.txt')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## N = 16"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th></th>\n",
" <th colspan=\"3\" halign=\"left\">memory</th>\n",
" <th colspan=\"5\" halign=\"left\">time</th>\n",
" </tr>\n",
" <tr>\n",
" <th></th>\n",
" <th>min</th>\n",
" <th>max</th>\n",
" <th>median</th>\n",
" <th>min</th>\n",
" <th>max</th>\n",
" <th>mean</th>\n",
" <th>sum</th>\n",
" <th>count</th>\n",
" </tr>\n",
" <tr>\n",
" <th>function</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Py_kgeneration</th>\n",
" <td>4.9</td>\n",
" <td>4.9</td>\n",
" <td>4.9</td>\n",
" <td>0.352931</td>\n",
" <td>0.352931</td>\n",
" <td>3.529310e-01</td>\n",
" <td>0.352931</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>generate</th>\n",
" <td>4.2</td>\n",
" <td>4.2</td>\n",
" <td>4.2</td>\n",
" <td>0.319565</td>\n",
" <td>0.319565</td>\n",
" <td>3.195650e-01</td>\n",
" <td>0.319565</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>gasdev</th>\n",
" <td>0.0</td>\n",
" <td>0.5</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.014654</td>\n",
" <td>3.830811e-05</td>\n",
" <td>0.156910</td>\n",
" <td>4096</td>\n",
" </tr>\n",
" <tr>\n",
" <th>fftma2</th>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>0.031217</td>\n",
" <td>0.031217</td>\n",
" <td>3.121700e-02</td>\n",
" <td>0.031217</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>covariance</th>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>0.030104</td>\n",
" <td>0.030104</td>\n",
" <td>3.010400e-02</td>\n",
" <td>0.030104</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>ran2</th>\n",
" <td>0.0</td>\n",
" <td>0.5</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000022</td>\n",
" <td>1.032080e-06</td>\n",
" <td>0.005437</td>\n",
" <td>5268</td>\n",
" </tr>\n",
" <tr>\n",
" <th>cov_value</th>\n",
" <td>0.0</td>\n",
" <td>0.5</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000013</td>\n",
" <td>8.762626e-07</td>\n",
" <td>0.003123</td>\n",
" <td>3564</td>\n",
" </tr>\n",
" <tr>\n",
" <th>fourt</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000269</td>\n",
" <td>0.000371</td>\n",
" <td>3.073333e-04</td>\n",
" <td>0.000922</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>cgrid</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000047</td>\n",
" <td>0.000047</td>\n",
" <td>4.700000e-05</td>\n",
" <td>0.000047</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>length</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000008</td>\n",
" <td>0.000010</td>\n",
" <td>8.666667e-06</td>\n",
" <td>0.000026</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>prebuild_gwn</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000023</td>\n",
" <td>0.000023</td>\n",
" <td>2.300000e-05</td>\n",
" <td>0.000023</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>build_real</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000021</td>\n",
" <td>0.000021</td>\n",
" <td>2.100000e-05</td>\n",
" <td>0.000021</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>clean_real</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000004</td>\n",
" <td>0.000004</td>\n",
" <td>4.000000e-06</td>\n",
" <td>0.000004</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>maxfactor</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000001</td>\n",
" <td>6.666667e-07</td>\n",
" <td>0.000002</td>\n",
" <td>3</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" memory time \\\n",
" min max median min max mean sum \n",
"function \n",
"Py_kgeneration 4.9 4.9 4.9 0.352931 0.352931 3.529310e-01 0.352931 \n",
"generate 4.2 4.2 4.2 0.319565 0.319565 3.195650e-01 0.319565 \n",
"gasdev 0.0 0.5 0.0 0.000000 0.014654 3.830811e-05 0.156910 \n",
"fftma2 1.0 1.0 1.0 0.031217 0.031217 3.121700e-02 0.031217 \n",
"covariance 1.0 1.0 1.0 0.030104 0.030104 3.010400e-02 0.030104 \n",
"ran2 0.0 0.5 0.0 0.000000 0.000022 1.032080e-06 0.005437 \n",
"cov_value 0.0 0.5 0.0 0.000000 0.000013 8.762626e-07 0.003123 \n",
"fourt 0.0 0.0 0.0 0.000269 0.000371 3.073333e-04 0.000922 \n",
"cgrid 0.0 0.0 0.0 0.000047 0.000047 4.700000e-05 0.000047 \n",
"length 0.0 0.0 0.0 0.000008 0.000010 8.666667e-06 0.000026 \n",
"prebuild_gwn 0.0 0.0 0.0 0.000023 0.000023 2.300000e-05 0.000023 \n",
"build_real 0.0 0.0 0.0 0.000021 0.000021 2.100000e-05 0.000021 \n",
"clean_real 0.0 0.0 0.0 0.000004 0.000004 4.000000e-06 0.000004 \n",
"maxfactor 0.0 0.0 0.0 0.000000 0.000001 6.666667e-07 0.000002 \n",
"\n",
" \n",
" count \n",
"function \n",
"Py_kgeneration 1 \n",
"generate 1 \n",
"gasdev 4096 \n",
"fftma2 1 \n",
"covariance 1 \n",
"ran2 5268 \n",
"cov_value 3564 \n",
"fourt 3 \n",
"cgrid 1 \n",
"length 3 \n",
"prebuild_gwn 1 \n",
"build_real 1 \n",
"clean_real 1 \n",
"maxfactor 3 "
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"analyze('log_16.txt')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## N = 32"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th></th>\n",
" <th colspan=\"3\" halign=\"left\">memory</th>\n",
" <th colspan=\"5\" halign=\"left\">time</th>\n",
" </tr>\n",
" <tr>\n",
" <th></th>\n",
" <th>min</th>\n",
" <th>max</th>\n",
" <th>median</th>\n",
" <th>min</th>\n",
" <th>max</th>\n",
" <th>mean</th>\n",
" <th>sum</th>\n",
" <th>count</th>\n",
" </tr>\n",
" <tr>\n",
" <th>function</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Py_kgeneration</th>\n",
" <td>2.7</td>\n",
" <td>2.7</td>\n",
" <td>2.7</td>\n",
" <td>1.180846</td>\n",
" <td>1.180846</td>\n",
" <td>1.180846e+00</td>\n",
" <td>1.180846</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>generate</th>\n",
" <td>4.9</td>\n",
" <td>4.9</td>\n",
" <td>4.9</td>\n",
" <td>0.840075</td>\n",
" <td>0.840075</td>\n",
" <td>8.400750e-01</td>\n",
" <td>0.840075</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>gasdev</th>\n",
" <td>-1.3</td>\n",
" <td>0.5</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.003243</td>\n",
" <td>1.513522e-05</td>\n",
" <td>0.495951</td>\n",
" <td>32768</td>\n",
" </tr>\n",
" <tr>\n",
" <th>fftma2</th>\n",
" <td>-2.2</td>\n",
" <td>-2.2</td>\n",
" <td>-2.2</td>\n",
" <td>0.338882</td>\n",
" <td>0.338882</td>\n",
" <td>3.388820e-01</td>\n",
" <td>0.338882</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>covariance</th>\n",
" <td>-2.5</td>\n",
" <td>-2.5</td>\n",
" <td>-2.5</td>\n",
" <td>0.330282</td>\n",
" <td>0.330282</td>\n",
" <td>3.302820e-01</td>\n",
" <td>0.330282</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>ran2</th>\n",
" <td>-0.3</td>\n",
" <td>0.5</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000103</td>\n",
" <td>9.740325e-07</td>\n",
" <td>0.040473</td>\n",
" <td>41552</td>\n",
" </tr>\n",
" <tr>\n",
" <th>cov_value</th>\n",
" <td>0.0</td>\n",
" <td>0.5</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000058</td>\n",
" <td>1.230344e-06</td>\n",
" <td>0.030296</td>\n",
" <td>24624</td>\n",
" </tr>\n",
" <tr>\n",
" <th>fourt</th>\n",
" <td>0.0</td>\n",
" <td>0.3</td>\n",
" <td>0.2</td>\n",
" <td>0.002212</td>\n",
" <td>0.003346</td>\n",
" <td>2.646333e-03</td>\n",
" <td>0.007939</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>prebuild_gwn</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000178</td>\n",
" <td>0.000178</td>\n",
" <td>1.780000e-04</td>\n",
" <td>0.000178</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>build_real</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000151</td>\n",
" <td>0.000151</td>\n",
" <td>1.510000e-04</td>\n",
" <td>0.000151</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>clean_real</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000104</td>\n",
" <td>0.000104</td>\n",
" <td>1.040000e-04</td>\n",
" <td>0.000104</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>cgrid</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000057</td>\n",
" <td>0.000057</td>\n",
" <td>5.700000e-05</td>\n",
" <td>0.000057</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>length</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000008</td>\n",
" <td>0.000016</td>\n",
" <td>1.300000e-05</td>\n",
" <td>0.000039</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>maxfactor</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000001</td>\n",
" <td>4.000000e-07</td>\n",
" <td>0.000002</td>\n",
" <td>5</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" memory time \\\n",
" min max median min max mean sum \n",
"function \n",
"Py_kgeneration 2.7 2.7 2.7 1.180846 1.180846 1.180846e+00 1.180846 \n",
"generate 4.9 4.9 4.9 0.840075 0.840075 8.400750e-01 0.840075 \n",
"gasdev -1.3 0.5 0.0 0.000000 0.003243 1.513522e-05 0.495951 \n",
"fftma2 -2.2 -2.2 -2.2 0.338882 0.338882 3.388820e-01 0.338882 \n",
"covariance -2.5 -2.5 -2.5 0.330282 0.330282 3.302820e-01 0.330282 \n",
"ran2 -0.3 0.5 0.0 0.000000 0.000103 9.740325e-07 0.040473 \n",
"cov_value 0.0 0.5 0.0 0.000000 0.000058 1.230344e-06 0.030296 \n",
"fourt 0.0 0.3 0.2 0.002212 0.003346 2.646333e-03 0.007939 \n",
"prebuild_gwn 0.0 0.0 0.0 0.000178 0.000178 1.780000e-04 0.000178 \n",
"build_real 0.0 0.0 0.0 0.000151 0.000151 1.510000e-04 0.000151 \n",
"clean_real 0.0 0.0 0.0 0.000104 0.000104 1.040000e-04 0.000104 \n",
"cgrid 0.0 0.0 0.0 0.000057 0.000057 5.700000e-05 0.000057 \n",
"length 0.0 0.0 0.0 0.000008 0.000016 1.300000e-05 0.000039 \n",
"maxfactor 0.0 0.0 0.0 0.000000 0.000001 4.000000e-07 0.000002 \n",
"\n",
" \n",
" count \n",
"function \n",
"Py_kgeneration 1 \n",
"generate 1 \n",
"gasdev 32768 \n",
"fftma2 1 \n",
"covariance 1 \n",
"ran2 41552 \n",
"cov_value 24624 \n",
"fourt 3 \n",
"prebuild_gwn 1 \n",
"build_real 1 \n",
"clean_real 1 \n",
"cgrid 1 \n",
"length 3 \n",
"maxfactor 5 "
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"analyze('log_32.txt')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## N = 64"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th></th>\n",
" <th colspan=\"3\" halign=\"left\">memory</th>\n",
" <th colspan=\"5\" halign=\"left\">time</th>\n",
" </tr>\n",
" <tr>\n",
" <th></th>\n",
" <th>min</th>\n",
" <th>max</th>\n",
" <th>median</th>\n",
" <th>min</th>\n",
" <th>max</th>\n",
" <th>mean</th>\n",
" <th>sum</th>\n",
" <th>count</th>\n",
" </tr>\n",
" <tr>\n",
" <th>function</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Py_kgeneration</th>\n",
" <td>233.5</td>\n",
" <td>233.5</td>\n",
" <td>233.5</td>\n",
" <td>30.354270</td>\n",
" <td>30.354270</td>\n",
" <td>30.354270</td>\n",
" <td>30.354270</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>generate</th>\n",
" <td>168.4</td>\n",
" <td>168.4</td>\n",
" <td>168.4</td>\n",
" <td>21.739521</td>\n",
" <td>21.739521</td>\n",
" <td>21.739521</td>\n",
" <td>21.739521</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>gasdev</th>\n",
" <td>-2.9</td>\n",
" <td>4.4</td>\n",
" <td>0.0</td>\n",
" <td>0.000009</td>\n",
" <td>0.000742</td>\n",
" <td>0.000061</td>\n",
" <td>16.076987</td>\n",
" <td>262144</td>\n",
" </tr>\n",
" <tr>\n",
" <th>fftma2</th>\n",
" <td>68.0</td>\n",
" <td>68.0</td>\n",
" <td>68.0</td>\n",
" <td>8.613854</td>\n",
" <td>8.613854</td>\n",
" <td>8.613854</td>\n",
" <td>8.613854</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>covariance</th>\n",
" <td>63.8</td>\n",
" <td>63.8</td>\n",
" <td>63.8</td>\n",
" <td>8.412335</td>\n",
" <td>8.412335</td>\n",
" <td>8.412335</td>\n",
" <td>8.412335</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>ran2</th>\n",
" <td>-0.6</td>\n",
" <td>0.8</td>\n",
" <td>0.0</td>\n",
" <td>0.000007</td>\n",
" <td>0.000281</td>\n",
" <td>0.000014</td>\n",
" <td>4.680977</td>\n",
" <td>333450</td>\n",
" </tr>\n",
" <tr>\n",
" <th>cov_value</th>\n",
" <td>-0.7</td>\n",
" <td>1.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000018</td>\n",
" <td>0.000320</td>\n",
" <td>0.000030</td>\n",
" <td>4.646094</td>\n",
" <td>156816</td>\n",
" </tr>\n",
" <tr>\n",
" <th>fourt</th>\n",
" <td>0.3</td>\n",
" <td>1.7</td>\n",
" <td>0.4</td>\n",
" <td>0.051830</td>\n",
" <td>0.078516</td>\n",
" <td>0.064563</td>\n",
" <td>0.193689</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>prebuild_gwn</th>\n",
" <td>2.5</td>\n",
" <td>2.5</td>\n",
" <td>2.5</td>\n",
" <td>0.002939</td>\n",
" <td>0.002939</td>\n",
" <td>0.002939</td>\n",
" <td>0.002939</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>build_real</th>\n",
" <td>-0.3</td>\n",
" <td>-0.3</td>\n",
" <td>-0.3</td>\n",
" <td>0.002089</td>\n",
" <td>0.002089</td>\n",
" <td>0.002089</td>\n",
" <td>0.002089</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>clean_real</th>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>1.0</td>\n",
" <td>0.001677</td>\n",
" <td>0.001677</td>\n",
" <td>0.001677</td>\n",
" <td>0.001677</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>cgrid</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000271</td>\n",
" <td>0.000271</td>\n",
" <td>0.000271</td>\n",
" <td>0.000271</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>length</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000048</td>\n",
" <td>0.000081</td>\n",
" <td>0.000065</td>\n",
" <td>0.000196</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>maxfactor</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000013</td>\n",
" <td>0.000015</td>\n",
" <td>0.000014</td>\n",
" <td>0.000056</td>\n",
" <td>4</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" memory time \\\n",
" min max median min max mean \n",
"function \n",
"Py_kgeneration 233.5 233.5 233.5 30.354270 30.354270 30.354270 \n",
"generate 168.4 168.4 168.4 21.739521 21.739521 21.739521 \n",
"gasdev -2.9 4.4 0.0 0.000009 0.000742 0.000061 \n",
"fftma2 68.0 68.0 68.0 8.613854 8.613854 8.613854 \n",
"covariance 63.8 63.8 63.8 8.412335 8.412335 8.412335 \n",
"ran2 -0.6 0.8 0.0 0.000007 0.000281 0.000014 \n",
"cov_value -0.7 1.0 0.0 0.000018 0.000320 0.000030 \n",
"fourt 0.3 1.7 0.4 0.051830 0.078516 0.064563 \n",
"prebuild_gwn 2.5 2.5 2.5 0.002939 0.002939 0.002939 \n",
"build_real -0.3 -0.3 -0.3 0.002089 0.002089 0.002089 \n",
"clean_real 1.0 1.0 1.0 0.001677 0.001677 0.001677 \n",
"cgrid 0.0 0.0 0.0 0.000271 0.000271 0.000271 \n",
"length 0.0 0.0 0.0 0.000048 0.000081 0.000065 \n",
"maxfactor 0.0 0.0 0.0 0.000013 0.000015 0.000014 \n",
"\n",
" \n",
" sum count \n",
"function \n",
"Py_kgeneration 30.354270 1 \n",
"generate 21.739521 1 \n",
"gasdev 16.076987 262144 \n",
"fftma2 8.613854 1 \n",
"covariance 8.412335 1 \n",
"ran2 4.680977 333450 \n",
"cov_value 4.646094 156816 \n",
"fourt 0.193689 3 \n",
"prebuild_gwn 0.002939 1 \n",
"build_real 0.002089 1 \n",
"clean_real 0.001677 1 \n",
"cgrid 0.000271 1 \n",
"length 0.000196 3 \n",
"maxfactor 0.000056 4 "
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"analyze('log_64.txt')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## N = 128"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"5897863\n"
]
},
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead tr th {\n",
" text-align: left;\n",
" }\n",
"\n",
" .dataframe thead tr:last-of-type th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr>\n",
" <th></th>\n",
" <th colspan=\"3\" halign=\"left\">memory</th>\n",
" <th colspan=\"5\" halign=\"left\">time</th>\n",
" </tr>\n",
" <tr>\n",
" <th></th>\n",
" <th>min</th>\n",
" <th>max</th>\n",
" <th>median</th>\n",
" <th>min</th>\n",
" <th>max</th>\n",
" <th>mean</th>\n",
" <th>sum</th>\n",
" <th>count</th>\n",
" </tr>\n",
" <tr>\n",
" <th>function</th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" <th></th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>Py_kgeneration</th>\n",
" <td>1864.7</td>\n",
" <td>1864.7</td>\n",
" <td>1864.7</td>\n",
" <td>126.767549</td>\n",
" <td>126.767549</td>\n",
" <td>126.767549</td>\n",
" <td>126.767549</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>generate</th>\n",
" <td>1759.0</td>\n",
" <td>1759.0</td>\n",
" <td>1759.0</td>\n",
" <td>97.731527</td>\n",
" <td>97.731527</td>\n",
" <td>97.731527</td>\n",
" <td>97.731527</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>gasdev</th>\n",
" <td>-134.6</td>\n",
" <td>18.5</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000772</td>\n",
" <td>0.000027</td>\n",
" <td>57.591066</td>\n",
" <td>2097152</td>\n",
" </tr>\n",
" <tr>\n",
" <th>fftma2</th>\n",
" <td>129.3</td>\n",
" <td>129.3</td>\n",
" <td>129.3</td>\n",
" <td>29.034783</td>\n",
" <td>29.034783</td>\n",
" <td>29.034783</td>\n",
" <td>29.034783</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>covariance</th>\n",
" <td>93.1</td>\n",
" <td>93.1</td>\n",
" <td>93.1</td>\n",
" <td>27.493769</td>\n",
" <td>27.493769</td>\n",
" <td>27.493769</td>\n",
" <td>27.493769</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>ran2</th>\n",
" <td>-4.8</td>\n",
" <td>2.2</td>\n",
" <td>0.0</td>\n",
" <td>0.000000</td>\n",
" <td>0.000065</td>\n",
" <td>0.000002</td>\n",
" <td>4.661066</td>\n",
" <td>2668394</td>\n",
" </tr>\n",
" <tr>\n",
" <th>cov_value</th>\n",
" <td>-0.6</td>\n",
" <td>0.7</td>\n",
" <td>0.0</td>\n",
" <td>0.000001</td>\n",
" <td>0.000281</td>\n",
" <td>0.000002</td>\n",
" <td>2.443765</td>\n",
" <td>1132300</td>\n",
" </tr>\n",
" <tr>\n",
" <th>fourt</th>\n",
" <td>-5.2</td>\n",
" <td>18.7</td>\n",
" <td>0.5</td>\n",
" <td>0.390409</td>\n",
" <td>0.591145</td>\n",
" <td>0.499473</td>\n",
" <td>1.498418</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>build_real</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.017085</td>\n",
" <td>0.017085</td>\n",
" <td>0.017085</td>\n",
" <td>0.017085</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>prebuild_gwn</th>\n",
" <td>17.0</td>\n",
" <td>17.0</td>\n",
" <td>17.0</td>\n",
" <td>0.014117</td>\n",
" <td>0.014117</td>\n",
" <td>0.014117</td>\n",
" <td>0.014117</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>clean_real</th>\n",
" <td>14.1</td>\n",
" <td>14.1</td>\n",
" <td>14.1</td>\n",
" <td>0.009017</td>\n",
" <td>0.009017</td>\n",
" <td>0.009017</td>\n",
" <td>0.009017</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>cgrid</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000187</td>\n",
" <td>0.000187</td>\n",
" <td>0.000187</td>\n",
" <td>0.000187</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>length</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000032</td>\n",
" <td>0.000035</td>\n",
" <td>0.000033</td>\n",
" <td>0.000099</td>\n",
" <td>3</td>\n",
" </tr>\n",
" <tr>\n",
" <th>maxfactor</th>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.0</td>\n",
" <td>0.000002</td>\n",
" <td>0.000003</td>\n",
" <td>0.000003</td>\n",
" <td>0.000008</td>\n",
" <td>3</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" memory time \\\n",
" min max median min max mean \n",
"function \n",
"Py_kgeneration 1864.7 1864.7 1864.7 126.767549 126.767549 126.767549 \n",
"generate 1759.0 1759.0 1759.0 97.731527 97.731527 97.731527 \n",
"gasdev -134.6 18.5 0.0 0.000000 0.000772 0.000027 \n",
"fftma2 129.3 129.3 129.3 29.034783 29.034783 29.034783 \n",
"covariance 93.1 93.1 93.1 27.493769 27.493769 27.493769 \n",
"ran2 -4.8 2.2 0.0 0.000000 0.000065 0.000002 \n",
"cov_value -0.6 0.7 0.0 0.000001 0.000281 0.000002 \n",
"fourt -5.2 18.7 0.5 0.390409 0.591145 0.499473 \n",
"build_real 0.0 0.0 0.0 0.017085 0.017085 0.017085 \n",
"prebuild_gwn 17.0 17.0 17.0 0.014117 0.014117 0.014117 \n",
"clean_real 14.1 14.1 14.1 0.009017 0.009017 0.009017 \n",
"cgrid 0.0 0.0 0.0 0.000187 0.000187 0.000187 \n",
"length 0.0 0.0 0.0 0.000032 0.000035 0.000033 \n",
"maxfactor 0.0 0.0 0.0 0.000002 0.000003 0.000003 \n",
"\n",
" \n",
" sum count \n",
"function \n",
"Py_kgeneration 126.767549 1 \n",
"generate 97.731527 1 \n",
"gasdev 57.591066 2097152 \n",
"fftma2 29.034783 1 \n",
"covariance 27.493769 1 \n",
"ran2 4.661066 2668394 \n",
"cov_value 2.443765 1132300 \n",
"fourt 1.498418 3 \n",
"build_real 0.017085 1 \n",
"prebuild_gwn 0.014117 1 \n",
"clean_real 0.009017 1 \n",
"cgrid 0.000187 1 \n",
"length 0.000099 3 \n",
"maxfactor 0.000008 3 "
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"analyze('log_128.txt')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## N = 256"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0\n",
"1\n",
"2\n",
"3\n",
"4\n",
"5\n",
"6\n",
"7\n",
"8\n",
"9\n"
]
}
],
"source": [
"dfs = []\n",
"for i in range(10):\n",
" print(i)\n",
" df = create_df(\"log_256_{}.txt\".format(i+1))\n",
" dfs.append(df)"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"functions = ['Py_kgeneration', 'generate', 'fftma2', 'covariance', 'gasdev', 'fourt', 'cov_value', 'ran2', 'build_real', 'prebuild_gwn', 'clean_real', 'cgrid', 'length', 'maxfactor']\n",
"\n",
"\n",
"#df_final = pd.concat(dfs).sort_values(by=('time', 'sum'), ascending=False) "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}